Luminya - AI-Powered Conversational Language Learning Systems with Interactive 3D Virtual Characters

Author

Soham Mestry, Swagat Kochrekar, Shubham Maldikar, Prof. Vaishali Yeole

Abstract

 While digital language education has expanded rapidly, bridging the gap between passive vocabulary acquisition and fluent spoken conversation remains a persistent pedagogical challenge. Recent developments in deep learning and web graphics make it possible to pair spoken dialogue systems with real-time, embodied virtual tutors directly in the browser. In this survey, we critically examine AI-driven conversational language learning systems, focusing on how interactive 3D virtual characters facilitate spoken dialogue practice. We synthesize progress across six foundational areas: Intelligent Computer-Assisted Language Learning (ICALL) frameworks, speech recognition and synthesis, large language models tailored for interactive tutoring, browser-based 3D avatar animation with speech-driven lip synchronization, cognitive spaced repetition scheduling, and natural language technologies for Indic languages. Our comparative analysis of ten leading academic and commercial platforms identifies an unresolved systemic deficit: current platforms either isolate conversational dialogue from visual embodiment, require cumbersome hardware, or overlook under-resourced Indic languages like Hindi and Marathi. To address these limitations, we evaluate architectural trade-offs, detail ongoing engineering hurdles—chiefly end-to-end latency and phonetic alignment—and propose a roadmap toward building accessible, browser-native conversational tutors.

Keywords

AI-Powered Language Learning, 3D Virtual Characters, Conversational AI, Speech Recognition, Adaptive Learning, Indian Languages, Lip Synchronization, Intelligent Tutoring Systems

Full Text:

Download Paper PDF

References

• Alkhatlan, A., & Kalita, J. (2018). Intelligent Tutoring Systems: A Comprehensive Historical Survey with Recent Developments. International Journal of Computer Applications, 181(43), 1–20.
• Baylor, A. L., & Kim, S. (2009). Designing Nonverbal Communication for Pedagogical Agents: When Less Is More. Computers in Human Behavior, 25(2), 450–457.
• Bione, T., Grimshaw, J., & Cardoso, W. (2017). An Evaluation of TTS as a Pedagogical Tool for Pronunciation Instruction: The 'Foreign' Language Context. In K. Borthwick, L. Bradley, & S. Thouësny (Eds.), CALL in a Climate of Change: Adapting to Turbulent Global Conditions – Short Papers from EUROCALL 2017 (pp. 56–61). Research-publishing.net.
• Brown, T., et al. (2020). Language Models Are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS), 33, 1877–1901.
• Cao, Y., Tien, W. C., Faloutsos, P., & Pighin, F. (2005). Expressive Speech-Driven Facial Animation. ACM Transactions on Graphics, 24(4), 1283–1302.
• Cardoso, W. (2018). Learning L2 Pronunciation with a Text-to-Speech Synthesizer. In P. Taalas, J. Jalkanen, L. Bradley, & S. Thouësny (Eds.), Future-Proof CALL: Language Learning as Exploration and Encounters – Short Papers from EUROCALL 2018 (pp. 43–48). Research-publishing.net.
• Cassell, J., Sullivan, J., Prevost, S., & Churchill, E. F. (Eds.). (2000). Embodied Conversational Agents. MIT Press.
• Cucchiarini, C., Strik, H., & Boves, L. (2000). Quantitative Assessment of Second Language Learners’ Fluency by Means of ASR Technology. Journal of the Acoustical Society of America, 107(2), 989–999.
• Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., & Black, M. J. (2019). Capture, Learning, and Synthesis of 3D Speaking Styles. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10101–10111.
• Doddapaneni, S., et al. (2023). Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 12402–12426.
• Edwards, P., Landreth, C., Fiume, E., & Singh, K. (2016). JALI: An Animator-Centric Viseme Model for Expressive Lip Synchronization. ACM Transactions on Graphics, 35(4), 127:1–127:11.
• Gaur, N., Farris, B., Haghani, P., Leal, I., Moreno, P. J., Prasad, M., Ramabhadran, B., & Zhu, Y. (2021). Mixture of Informed Experts for Multilingual Speech Recognition. IEEE ICASSP, 6234–6238.
• Gratch, J., Wang, N., Gerten, J., Fast, E., & Duffy, R. (2007). Creating Rapport with Virtual Agents. In C. Pelachaud et al. (Eds.), Intelligent Virtual Agents (IVA 2007), Lecture Notes in Computer Science (Vol. 4722, pp. 125–138). Springer, Berlin, Heidelberg.
• Gupta, A., & Gaur, S. (2019). Hindi Text-to-Speech System Using Deep Learning. International Conference on Communication and Electronics Systems (ICCES), 1297–1301.
• Han, J., Zhao, W., & Chen, L. (2024). LLM-Based Conversational AI Tutors for Language Learning: A Design Framework. Proc. ACM Learning at Scale (L@S), 45–56.
• Hansen, M. (2023). Piper: A Fast, Local Neural Text to Speech System [Computer software]. GitHub. https://github.com/rhasspy/piper
• Heidig, S., & Clarebout, G. (2011). Do Pedagogical Agents Make a Difference to Student Motivation and Learning? Educational Research Review, 6(1), 27–54.
• Huang, X., Zou, D., Cheng, G., & Xie, H. (2021). A Systematic Review of AR and VR Enhanced Language Learning. Sustainability, 13(9), 4639.
• Javed, T., Bhogale, K., Raman, A., Kumar, P., Kunchukuttan, A., & Khapra, M. M. (2023). IndicSUPERB: A Speech Processing Universal Performance Benchmark for Indian Languages. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 12942–12950.
• Javed, T., et al. (2024). IndicVoices: Towards Building an Inclusive Multilingual Speech Dataset for Indian Languages. Findings of the Association for Computational Linguistics: ACL 2024, 10740–10782.
• Ji, H., & Park, M. (2023). The Effects of Embodied Conversational Agents on Foreign Language Anxiety and Speaking Competence. Computers & Education, 194, 104698.
• Johnson, W. L., & Lester, J. C. (2016). Face-to-Face Interaction with Pedagogical Agents, Twenty Years Later. International Journal of AI in Education, 26(1), 25–36.
• Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The State and Fate of Linguistic Diversity and Inclusion in the NLP World. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 6282–6293.
• Kakwani, D., et al. (2020). IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages. Findings of EMNLP 2020, 4948–4961.
• Kannan, J., & Munday, P. (2018). New Trends in Second Language Learning and Teaching through the Lens of ICT, Networked Learning, and Artificial Intelligence. Círculo de Lingüística Aplicada a la Comunicación, 76, 13–30.
• Kasneci, E., et al. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. Learning and Individual Differences, 103, 102274.
• Kim, I. S. (2006). Automatic Speech Recognition: Reliability and Pedagogical Implications for Teaching Pronunciation. Educational Technology & Society, 9(1), 322–334.
• Kim, J., Kong, J., & Son, J. (2021). Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech. Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139, 5530–5540.
• Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for Language Teaching and Learning. RELC Journal, 54(2), 537–550.
• Kunchukuttan, A. (2020). The IndicNLP Library. AI4Bharat Open Research Initiative. https://github.com/anoopkunchukuttan/indic_nlp_library
• Mai, Z., & Carson-Berndsen, J. (2024). Enhancing Conversational Quality in Language Learning Chatbots: An Evaluation of GPT4 for ASR Error Correction. IEEE ICASSP, 11001–11005.
• Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., & Cao, T. H. (2018). Chatbot as an Intelligent Personal Assistant for Mobile Language Learning. Proceedings of the 2018 2nd International Conference on Education and E-Learning (ICEEL 2018), 16–21.
• Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. Proceedings of ICML, 28492–28518.
• Reddy, S., Labutov, I., Banerjee, S., & Joachims, T. (2016). Unbounded Human Learning: Optimal Scheduling for Spaced Repetition. Proceedings of ACM SIGKDD, 1815–1824.
• Settles, B., & Meeder, B. (2016). A Trainable Spaced Repetition Model for Language Learning. Proceedings of ACL, 1848–1858.
• Settles, B., Brust, C., Gustafson, E., Hagiwara, T., & Madnani, N. (2018). Second Language Acquisition Modeling. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2018, NAACL-HLT), 56–65.
• Tabibian, B., Upadhyay, U., De, A., Zarezade, A., Schölkopf, B., & Gomez-Rodriguez, M. (2019). Enhancing Human Learning via Spaced Repetition Optimization. PNAS, 116(10), 3988–3993.
• Tan, X., Qin, T., Soong, F., & Liu, T. Y. (2021). A Survey on Neural Speech Synthesis. arXiv preprint arXiv:2106.15561.
• Taylor, S., et al. (2017). A Deep Learning Approach for Generalized Speech Animation. ACM Transactions on Graphics (SIGGRAPH), 36(4), 93:1–93:11.
• Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971.
• Vandewaetere, M., & Desmet, P. (2012). Introduction to the Special Issue on Adapting in CALL. Computer Assisted Language Learning, 25(5), 399–404.
• Wolf, D. S. (2023). Rhubarb Lip Sync: Automated Lip-Sync Tool for 2D and 3D Characters [Computer software]. GitHub. https://github.com/DanielSWolf/rhubarb-lip-sync
• Wozniak, P. A., & Gorzelanczyk, E. J. (1994). Optimization of Repetition Spacing in the Practice of Learning. Acta Neurobiologiae Experimentalis, 54, 59–62.
• Yan, L., et al. (2024). Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review. British Journal of Educational Technology, 55(1), 90–112.
• Ye, J., Su, J., & Cao, Y. (2022). A Stochastic Shortest Path Algorithm for Optimizing Spaced Repetition Scheduling. Proceedings of ACM SIGKDD, 4381–4390.

Share your valuable work from Social Media Buttons