Analysis of Environmental Factors Affecting Real-Time Hand Gesture Recognition Using MediaPipe Landmarks

Author

Smita Mangesh Junnarkar, Rohit Bose

Abstract

 Touch-free control through hand gestures looks convincing in a demonstration, yet many prototypes stumble as soon as they leave the room in which they were built. This paper investigates why. It studies gesture recognition systems that pass MediaPipe hand landmarks to a small classifier, and it follows the route by which a disturbance in the scene travels from the camera, through landmark estimation, to the final gesture label. Eight families of disturbance are examined: lighting, background, distance, camera hardware, viewpoint, occlusion, motion together with frame rate, and differences between users and devices. For each family the paper explains the mechanism, the symptom a developer is likely to notice, and a practical remedy. A test protocol is then proposed, with condition levels, normalisation and stability formulas, and reporting metrics, so that other groups can repeat the measurements on their own equipment. The work is analytical and reports no new benchmark numbers. Its purpose is to help researchers and students plan fair experiments and build gesture interfaces that stay dependable in everyday rooms.

Keywords

Hand gesture recognition, MediaPipe, hand landmarks, environmental factors, illumination, robustness, real-time systems, human-computer interaction

Full Text:

Download Paper PDF

References

[1] C. Lugaresi et al., “MediaPipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019.

[2] F. Zhang et al., “MediaPipe Hands: On-device real-time hand tracking,” arXiv preprint arXiv:2006.10214, 2020.

[3] T. Simon, H. Joo, I. Matthews, and Y. Sheikh, “Hand keypoint detection in single images using multiview bootstrapping,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1145–1153.

[4] S. Mitra and T. Acharya, “Gesture recognition: A survey,” IEEE Trans. Systems, Man, and Cybernetics, Part C, vol. 37, no. 3, pp. 311–324, 2007.

[5] S. S. Rautaray and A. Agrawal, “Vision based hand gesture recognition for human computer interaction: A survey,” Artificial Intelligence Review, vol. 43, no. 1, pp. 1–54, 2015.

[6] P. K. Pisharady and M. Saerbeck, “Recent methods and databases in vision-based hand gesture recognition: A review,” Computer Vision and Image Understanding, vol. 141, pp. 152–165, 2015.

[7] A. G. Howard et al., “MobileNets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.

[8] C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995.

[9] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.

[10] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.

[11] G. Bradski, “The OpenCV Library,” Dr. Dobb's Journal of Software Tools, 2000.

[12] V. Bazarevsky et al., “BlazePose: On-device real-time body pose tracking,” arXiv preprint arXiv:2006.10204, 2020.

[13] T. B. Moeslund and E. Granum, “A survey of computer vision-based human motion capture,” Computer Vision and Image Understanding, vol. 81, no. 3, pp. 231–268, 2001.

[14] J. Buolamwini and T. Gebru, “Gender shades: Intersectional accuracy disparities in commercial gender classification,” Proceedings of Machine Learning Research, vol. 81, pp. 77–91, 2018.

[15] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.

Share your valuable work from Social Media Buttons