Real-Time Hands-Free Human-Computer Interaction: An Optical Gaze, Wink, and Facial Landmark Telemetry Mouse System

Author

V. Youvakishore, K. Balakrishna Maruthiram

Abstract

 Human-computer interaction (HCI) remains heavily reliant on physical input peripherals such as mechanical mice, trackpads, and keyboards. For individuals afflicted with severe motor impairments, quadriplegia, amyotrophic lateral sclerosis (ALS), spinal cord injuries, or repetitive strain trauma, conventional physical interfaces present insurmountable accessibility barriers. Existing vision-based assistive pointing technologies frequently demand proprietary infrared illuminators, computationally burdensome deep learning models requiring discrete graphics hardware, or extensive per-user calibration procedures. Furthermore, webcam-based gaze trackers suffer persistently from three operational dilemmas: high-frequency cursor jitter induced by microscopic involuntary eye and head tremors, target reachability constraints along display perimeters, and unintended click actuations caused by involuntary bilateral blinking (the classic "Midas Touch" problem).

To overcome these constraints, this research presents an end-to-end, real-time, non-invasive hands-free pointing and interaction system operating strictly on commodity monocular RGB webcams without dedicated hardware accelerators. Leveraging the Google MediaPipe FaceMesh framework, the architecture extracts 468 three-dimensional facial geometric landmarks alongside refined iris keypoints in real time. We formulate a centroid-relative spatial mapping algorithm that computes iris displacements with respect to the mathematical center of the acquired frame, effectively eliminating baseline ocular drift and asymmetric bias. To suppress high-frequency micro-tremors without introducing unacceptable phase latency, a rolling-buffer temporal averaging filter (N=12 frames) is integrated with a non-linear 2.5×radial sensitivity scalar, enabling effortless edge-to-edge display navigation while requiring minimal cervical rotation. Click execution is governed by a temporal state engine that analyzes vertical eyelid aperture differentials between landmarks 374 and 386; intentional winks are isolated from involuntary bilateral blinks via a dual-eye comparative filter, yielding a dramatic reduction in accidental click activations. Furthermore, to address vertical navigation in long-form digital content, we propose an oromandibular gesture mode: monitoring Euclidean lip separation (landmarks 13 and 14) dynamically toggles an automatic document scrolling engine whose directional polarity is indexed by the cursor's vertical position relative to the screen midpoint. Evaluated on commodity consumer hardware, the system maintains a deterministic throughput of 15–25 frames per second (FPS) on a multi-core CPU architecture with an overall click precision exceeding 90% and stable zero-drift boundary clamping. The resulting framework provides a robust, zero-cost, and computationally efficient assistive tool that democratizes digital accessibility for motor-impaired individuals.

Keywords

Assistive Technology, Human-Computer Interaction (HCI), Computer Vision, MediaPipe FaceMesh, Iris Landmark Telemetry, Gaze Estimation, Eye Aspect Ratio, Wink Click Discrimination, Oromandibular Scrolling, PyAutoGUI, OpenCV, Real-Time Edge Processing.

Full Text:

Download Paper PDF

References

[1] G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library, 1st ed. Sebastopol, CA: O’Reilly Media, 2008.

[2] C. Lugaresi, J. Tang, H. Hadon, C. Wang, M. Uboweja, V. Bazarevsky, S. Fan, K. Guo, N. El-Kazak, M. Tseng, S. Burkov, J. Shen, M. Yuan, D. Lin, S. Zhang, G. Armstrong, and M. Grundmann, "MediaPipe: A Framework for Building Perception Pipelines," arXiv preprint arXiv:1906.08172, 2019.

[3] A. Sweigart, "PyAutoGUI Documentation: Cross-platform GUI automation Python module for human beings," 2020. [Online]. Available: https://pyautogui.readthedocs.io/

[4] Python Software Foundation, "Python Language Reference, Version 3.12," 2024. [Online]. Available: https://www.python.org/

[5] V. Kazemi and J. Sullivan, "One Millisecond Face Alignment with an Ensemble of Regression Trees," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, 2014, pp. 1867–1874.

[6] P. Viola and M. Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features," in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Kauai, HI, 2001, pp. 511–518.

[7] A. Kar, "Facial Mesh and Iris Tracking with MediaPipe: State of the Art Real-Time On-Device Computer Vision," Google AI Blog, 2020.

[8] T. Soukupová and J. Čech, "Real-Time Eye Blink Detection using Facial Landmarks," in Proceedings of the 21st Computer Vision Winter Workshop (CVWW), Rimske Toplice, Slovenia, 2016, pp. 1–8.

[9] R. J. K. Jacob, "The Use of Eye Movements in Human-Computer Interaction Techniques: What You Look At is What You Get," ACM Transactions on Information Systems (TOIS), vol. 9, no. 3, pp. 152–169, 1991.

[10] A. T. Duchowski, Eye Tracking Methodology: Theory and Practice, 3rd ed. London, U.K.: Springer-Verlag, 2017.

[11] K. Holmqvist, M. Nyström, R. Andersson, R. Dewhurst, H. Jarodzka, and K. Van de Weijer, Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford, U.K.: Oxford University Press, 2011.

[12] D. W. Hansen and X. Qiang, "In the Eye of the Beholder: A Survey of Models for Eyes and Gaze," IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 32, no. 3, pp. 478–500, 2010.

[13] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C. L. Chang, and M. Grundmann, "MediaPipe Hands: On-Device Real-Time Hand Tracking," arXiv preprint arXiv:2006.10214, 2020.

[14] I. MacKenzie, Human-Computer Interaction: An Empirical Research Perspective. Waltham, MA: Morgan Kaufmann, 2013.

[15] P. Majaranta and A. Bulling, "Eye Tracking and Eye-Based Human-Computer Interaction," in Advances in Physiological Computing, S. Fairclough and K. Gilleade, Eds. London: Springer, 2014, pp. 39–65.

Share your valuable work from Social Media Buttons