A MACHINE LEARNING BASED REAL-TIME SIGN LANGUAGE RECOGNITION SYSTEM WITH SPEECH CONVERSION

Author

C. Fairlin Gold Teena , Dr. J. Evangelin Deva Sheela

Abstract

 Sign language provides an important means of communication for people with hearing and speech disabilities. However, communication between sign language users and people who do not understand sign language can be challenging. This paper presents a real-time sign language recognition system using computer vision and machine learning techniques. The proposed system uses a webcam to capture hand gestures and MediaPipe to detect 21 hand landmarks from the left and right hands. The extracted landmark coordinates are used as numerical features for classification. A dataset containing 4,000 samples of 20 static word-level gestures was prepared for the system. Three machine learning algorithms, namely K-Nearest Neighbors (KNN), Support Vector Machine (SVM), and Random Forest (RF), were implemented and evaluated using Accuracy, Precision, Recall, and F1-Score. The experimental results show that SVM achieved the highest offline accuracy of 96.36%. In real-time testing, SVM achieved an accuracy of 96.5%. The recognized gestures are displayed as text and converted into speech, providing an accessible communication interface. The proposed system demonstrates the potential of machine learning and computer vision for real-time sign language communication assistance.

Keywords

Sign Language Recognition, Machine Learning, MediaPipe, Hand Landmark Detection, KNN, SVM, Random Forest, Computer Vision, Speech Conversion, Real-Time Recognition

Full Text:

Download Paper PDF

References

The following references are supported by the research papers you uploaded and are suitable for this journal.

[1] D. KhajehPourian and A. D. Zarei, “Sign Language Recognition using Modified Deep Learning Network: CNNSa-LSTM,” Scientific Reports, vol. 14, 2024.

 [2] H. Hu et al., “SignBERT: Pre-Training of Hand-Model-Aware Representation for Sign Language Recognition,” arXiv preprint arXiv:2110.05382, 2021.

 [3] I. A. Adeyanju, O. O. Bello, and M. A. Adegboye, “Machine Learning Methods for Sign Language Recognition: A Critical Review and Analysis,” Intelligent Systems with Applications, vol. 12, 2021, Art. no. 200056.

[4] K. Bantupalli and Y. Xie, “American Sign Language Recognition using Deep Learning and Computer Vision,” 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA, 2018, pp. 4896–4899.

[5] K. Shenoy, T. Dastane, V. Rao, and D. Vyavaharkar, “Real-time Indian Sign Language (ISL) Recognition,” arXiv preprint arXiv:2108.10970, 2021.

[6]  N. Gopinath, J. Anuja, S. Anusha, and V. Monisha, “A Survey on Hand Gesture Recognition Using Machine Learning,” International Research Journal of Engineering and Technology (IRJET), 2020.

[7]  N. Rajasekhar, M. G. Yadav, C. Vedantam, K. Pellakuru, and C. Navapete, “Sign Language Recognition using Machine Learning Algorithm,” 2023 International Conference on Sustainable Computing and Smart Systems (ICSCSS), Coimbatore, India, 2023, pp. 303–306.

[8]  O. Vedak, P. Zavre, A. Todkar, and M. Patil, “Sign Language Interpreter using Image Processing and Machine Learning,” International Research Journal of Engineering and Technology (IRJET), 2019.Top of FormBottom of Form

[9]  P. Navendu and V. Sahula, “Word Level Sign Language Recognition using MediaPipe and LSTM-GRU Network,” TechRxiv, July 2024.

[10] V. Sarma Ch, H. Bindu Gogineni, B. Prasad, and P. Praveen Kumar, “Real-Time Recognition of American Sign Language Using Mediapipe and Machine Learning Techniques,” Journal of Information Systems Engineering and Management, vol. 10, no. 3, 2025, pp. 1–15.

Share your valuable work from Social Media Buttons