LR.AI: Lip Reading AI

Author

Swapnil P. Suryavanshi, Kaustubh Mahesh Kulkarni, Varad Manoj Kulkarni

Abstract

 Human communication largely depends on speech, but traditional speech recognition systems fail in noisy environments or for people with hearing or speech impairments [1], [3]. This project proposes an AI-based Lip-Reading Assistant that interprets spoken words by analyzing visual lip movements instead of relying on audio. The system uses Computer Vision and Deep Learning techniques to extract lip-region features from video frames and translate them into text in real time.

The system uses Google MediaPipe to track 21 specific points on the lips and an LSTM (Long Short-Term Memory) neural network to turn those movements into words. The project is built with a Python (Flask) backend that does the heavy AI work and a React frontend that shows the live video to the user. These two parts talk to each other instantly using WebSockets. The final result is a fast, easy-to-use tool.

The proposed system aims to assist individuals with hearing or speech disabilities, improve accessibility in communication devices, and enhance voice free interaction in noisy or privacy sensitive environments. Through real-time video processing and efficient deep learning models, this project demonstrates how AI-driven visual speech recognition can bridge communication gaps between humans and machines.

Keywords

Lip Reading; Computer Vision; Deep Learning; LSTM (Long Short-Term Memory); Google MediaPipe; Assistive Technology; Real-Time Speech Recognition; Motion Gating; Human-Computer Interaction

Full Text:

Download Paper PDF

References

[1] M. Abishek, S. Harish, R. Krishna Prasath, D. Sudarshan, and P. Supriya, “Deep Learning Based Lip Reading for Speech Recognition,” in 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), 2024, doi: 10.1109/ICCCNT61001.2024.10725052.

[2] H. Kode, B. S. S. Kotipalli, H. R. Cheruku, and S. S. Gadde, “A Survey on Deep Learning-Based Approaches for Automated Lip Reading,” in 2025 IEEE 2nd International Conference on Deep Learning and Computer Vision (DLCV), 2025, doi: 10.1109/DLCV65218.2025.11088852.

[3] S. Fenghour, D. Chen, K. Guo, B. Li, and P. Xiao, “Deep Learning-Based Automated Lip-Reading: A Survey,” IEEE Access, vol. 9, pp. 121184–121205, 2021, doi: 10.1109/ACCESS.2021.3107946.

[4] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep Audio-Visual Speech Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 12, pp. 8717–8727, 2022, doi: 10.1109/TPAMI.2018.2889052.

[5] A. Paszke et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in Advances in Neural Information Processing Systems 32 (NeurIPS), pp. 8024–8035, 2019.

[6] D. Ivanko, E. Ryumina, and D. Ryumin, “Improved Automatic Lip-Reading Based on the Evaluation of Intensity Level of Speaker’s Emotion,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLVIII-2/W3-2023, pp. 89–94, 2023, doi: 10.5194/isprs-archives-XLVIII-2-W3-2023-89-2023.

[7] Divyaprabha, S. S. T. V. Sagar, M. G., G. C. Reddy, and M. K. M. O., “Lip Reading Using CNN and Bi-LSTM,” International Journal of Creative Research Thoughts (IJCRT), vol. 12, no. 6, pp. c721–c729, Jun. 2024.

[8] J. E. Jeyaswamidoss, E. Y. R. Ravichandran, H. A. R. Ahamed, P. S. Ananthi, and S. Devarajan, “Silent Speech Recognition: Lip Reading Model,” AIP Conference Proceedings, vol. 3279, 020026, 2025, doi: 10.1063/5.0262876.

Share your valuable work from Social Media Buttons