HAND GESTURE DETECTION AND RECOGNITION USING A CONVOLUTIONAL NEURAL NETWORK

Author

THIRISHA.V

Abstract

 RGB hand gesture recognition can support touchless controls, but a classifier evaluated only on manually cropped images does not show how often a webcam system will miss a gesture. This report proposes a two-stage pipeline that uses MediaPipe Hand Landmarker to locate hands and a compact convolutional neural network (CNN) to classify RGB hand crops. The proposed study uses the subject-disjoint splits of the original HaGRID benchmark to evaluate 18 static gestures and an additional class for visible hands without a target gesture. Two tests separate classification on annotated crops from end-to-end recognition on full images. The first uses macro F1 to summarize performance across classes; the second requires both a correct gesture label and adequate overlap with the reference hand box. A separate webcam protocol measures processing latency, throughput, and false activations in idle scenes. Published benchmark results establish context for the design. The proposed pipeline has not yet been evaluated, so its accuracy, operating speed, and suitability for live use remain open questions.

Keywords

hand gesture detection; static gesture recognition; convolutional neural network; MediaPipe; HaGRID; human–computer interaction.

Full Text:

Download Paper PDF

References

[1] W. T. Freeman and M. Roth, “Orientation Histograms for Hand Gesture Recognition,” Mitsubishi Electric Research Laboratories, Tech. Rep. TR94-03, 1994. https://www.merl.com/publications/TR94-03

[2] R. F. Pinto Jr., C. D. B. Borges, A. M. A. Almeida, and I. C. Paula Jr., “Static Hand Gesture Recognition Based on Convolutional Neural Networks,” Journal of Electrical and Computer Engineering, vol. 2019, Art. no. 4167890, 2019. doi:10.1155/2019/4167890.

[3] F. Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking,” arXiv:2006.10214, 2020. https://arxiv.org/abs/2006.10214

[4] A. Kapitanov, K. Kvanchiani, A. Nagaev, R. Kraynov, and A. Makhliarchuk, “HaGRID – HAnd Gesture Recognition Image Dataset,” in Proc. IEEE/CVF Winter Conf. Applications of Computer Vision, 2024, pp. 4560–4569. doi:10.1109/WACV57701.2024.00451.

[5] A. Howard et al., “Searching for MobileNetV3,” in Proc. IEEE/CVF Int. Conf. Computer Vision, 2019, pp. 1314–1324. doi:10.1109/ICCV.2019.00140.

[6] P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz, “Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016, pp. 4207–4215. doi:10.1109/CVPR.2016.456.

[7] HaGRID project, “HAnd Gesture Recognition Image Dataset,” 18-class release and annotations. https://github.com/hukenovs/hagrid/tree/Hagrid_v2

[8] Google AI Edge, “Hand landmarks detection guide.” https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker

Share your valuable work from Social Media Buttons