Analysis of Real-Time Speech Transcription Models

Author

Susmita Sahu, Shruti Patel, Nidhi Sinha, Deepti Chandran

Abstract

 Real-time speech transcription turns spoken language into text as it is spoken. It is the basis of live captioning, lecture and meeting notes, dictation and voice interfaces, and it depends on automatic speech recognition (ASR) that stays accurate across languages, accents and mixed-language speech with low delay. This paper analyses forty-one studies published between 2022 and 2026 that bear on building and adapting such models. All were identified through Google Scholar and are grouped into themes: massively multilingual foundation models, Whisper adaptation, parameter-efficient fine-tuning and adapters, streaming architectures, voice activity detection, language identification, speaker diarization, speech translation, and lecture captioning and evaluation. Eighteen studies concern Indian languages and twelve concern other languages or very large multilingual systems. A single comparative table, built only from what the published abstracts state, lists the model, approach, languages and key finding of every study. No single model is best on every measure: the Universal Speech Model, an industrial Conformer/RNN-T system and the Stateful FastConformer stand out for efficiency, while adapters and LoRA stand out for low-cost adaptation. In the reviewed abstracts, spontaneous speech and accuracy under streaming are reported as difficult, and none describes joint handling of language switching, speaker changes and latency. The paper closes with research gaps and future directions, including unified evaluation of accuracy and delay, dialect-aware adaptation and classroom-oriented benchmarks.

Keywords

automatic speech recognition, fine-tuning, real-time transcription, multilingual ASR, code-switching, Indian languages, Whisper, parameter-efficient adaptation, streaming ASR, speech translation

Full Text:

Download Paper PDF

References

[1] S. Pal, V. Badhe, and C. Dasgupta, “Investigating the role of closed captioning and live transcription on DHH students’ perception of inclusivity in a face-to-face classroom environment,” in Proc. IEEE Int. Conf. Adv. Learn. Technol. (ICALT), 2023, doi: 10.1109/icalt58122.2023.00070.

[2] K. Kuhn et al., “Measuring the accuracy of automatic speech recognition solutions,” ACM Trans. Access. Comput., 2023, doi: 10.1145/3636513.

[3] J. Park et al., “A low-latency streaming on-device automatic speech recognition system using a CNN acoustic model on FPGA and a language model on smartphone,” Electronics, 2022, doi: 10.3390/electronics11121831.

[4] S. Ding et al., “Personal VAD 2.0: Optimizing personal voice activity detection for on-device speech recognition,” in Proc. Interspeech, 2022, doi: 10.21437/interspeech.2022-856.

[5] R. Dabre and H. Song, “NICT’s cascaded and end-to-end speech translation systems using Whisper and IndicTrans2 for the Indic task,” in Proc. 21st Int. Conf. Spoken Lang. Transl. (IWSLT), 2024, doi: 10.18653/v1/2024.iwslt-1.3.

[6] V. Mujadia and D. Sharma, “Towards speech to speech machine translation focusing on Indian languages,” in Proc. 17th Conf. Eur. Chapter Assoc. Comput. Linguistics: Syst. Demonstrations, 2023, doi: 10.18653/v1/2023.eacl-demo.19.

[7] A. Biswas et al., “Adapting Whisper for low-resource Hindi-English code-mix speech with on-the-fly augmentation & LLM-synthesised data,” in Proc. Interspeech, 2025, doi: 10.21437/interspeech.2025-447.

[8] A. Antony et al., “An improved transformer transducer architecture for Hindi-English code switched speech recognition,” in Proc. Interspeech, 2022, doi: 10.21437/interspeech.2022-10763.

[9] V. Noroozi et al., “Stateful Conformer with cache-based inference for streaming automatic speech recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2024, doi: 10.1109/icassp48485.2024.10446861.

[10] Y. Liu, X. Yang, and D. Qu, “Exploration of Whisper fine-tuning strategies for low-resource ASR,” EURASIP J. Audio, Speech, Music Process., 2024, doi: 10.1186/s13636-024-00349-3.

[11] Y. Zhang et al., “Google USM: Scaling automatic speech recognition beyond 100 languages,” 2023, arXiv:2303.01037.

[12] B. Thomas, S. Kessler, and S. Karout, “Efficient adapter transfer of self-supervised speech models for automatic speech recognition,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2022, doi: 10.1109/icassp43922.2022.9746223.

[13] S. C. Bajaj et al., “Large vocabulary continuous speech recognition system for Marathi,” in Proc. Int. Conf. Adv. Electron., Commun., Comput. Intell. Inf. Syst. (ICAECIS), 2023, doi: 10.1109/icaecis58353.2023.10170653.

[14] C. H. Nga et al., “Cyclic transfer learning for Mandarin-English code-switching speech recognition,” IEEE Signal Process. Lett., 2023, doi: 10.1109/LSP.2023.3307350.

[15] H. Polat et al., “Implementation of a Whisper architecture-based Turkish automatic speech recognition (ASR) system and evaluation of the effect of fine-tuning with a low-rank adaptation (LoRA) adapter on its performance,” Electronics, 2024, doi: 10.3390/electronics13214227.

[16] B. Li et al., “Efficient domain adaptation for speech foundation models,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2023, doi: 10.1109/icassp49357.2023.10096330.

[17] V. S. Lodagala et al., “All Ears: Building self-supervised learning based ASR models for Indian languages at scale,” in Proc. Interspeech, 2024, doi: 10.21437/interspeech.2024-2520.

[18] J. Zhao and W.-Q. Zhang, “Improving automatic speech recognition performance for low-resource languages with self-supervised models,” IEEE J. Sel. Topics Signal Process., 2022, doi: 10.1109/jstsp.2022.3184480.

[19] SEAMLESS Communication Team, L. Barrault et al., “Joint speech and text machine translation for up to 100 languages,” Nature, 2025, doi: 10.1038/s41586-024-08359-z.

[20] Y. Peng et al., “Reproducing Whisper-style training using an open-source toolkit and publicly available data,” in Proc. IEEE Autom. Speech Recognit. Understand. Workshop (ASRU), 2023, doi: 10.1109/asru57964.2023.10389676.

[21] A. Tjandra et al., “Massively multilingual ASR on 70 languages: Tokenization, architecture, and generalization capabilities,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), 2023, doi: 10.1109/icassp49357.2023.10094667.

[22] F. McCann Ramirez et al., “Anatomy of industrial scale multilingual ASR,” 2024, arXiv:2404.09841, doi: 10.48550/arXiv.2404.09841.

[23] Ö. T. Özyilmaz, M. Coler, and M. Valdenegro-Toro, “Overcoming data scarcity in multi-dialectal Arabic ASR via Whisper fine-tuning,” in Proc. Interspeech, 2025, doi: 10.21437/Interspeech.2025-2260.

[24] A. K. Sharma, M. Pandya, and A. Shukla, “Fine-tuning Whisper Tiny for Swahili ASR: Challenges and recommendations for low-resource speech recognition,” in Proc. 6th Workshop Afr. Natural Lang. Process. (AfricaNLP), 2025, doi: 10.18653/v1/2025.africanlp-1.11.

[25] J. Kunisetty et al., “Advancing ASR for Indian-accented English: Dataset creation and Whisper fine-tuning,” Procedia Comput. Sci., 2025, doi: 10.1016/j.procs.2025.04.513.

[26] K. Swapna et al., “Multilingual ASR for Indic languages using OpenAI’s Whisper,” in Proc. 2nd Int. Conf. Circuits, Power Intell. Syst. (CCPIS), 2025, doi: 10.1109/ccpis65231.2025.11234204.

[27] N. Inoue et al., “ELP-adapters: Parameter efficient adapter tuning for various speech processing tasks,” IEEE/ACM Trans. Audio, Speech, Lang. Process., 2024, doi: 10.1109/taslp.2024.3434445.

[28] K.-M. Lyu, R.-y. Lyu, and H.-T. Chang, “Real-time multilingual speech recognition and speaker diarization system based on Whisper segmentation,” PeerJ Comput. Sci., 2024, doi: 10.7717/peerj-cs.1973.

[29] H. Liu et al., “Efficient self-supervised learning representations for spoken language identification,” IEEE J. Sel. Topics Signal Process., 2022, doi: 10.1109/jstsp.2022.3201445.

[30] K. Dhawan, D. Rekesh, and B. Ginsburg, “Unified model for code-switching speech recognition and language identification based on concatenated tokenizer,” in Proc. 6th Workshop Comput. Approaches Linguistic Code-Switching, 2023, doi: 10.18653/v1/2023.calcs-1.7.

[31] H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: Principle, benchmark, and recipe,” in Proc. Interspeech, 2023, doi: 10.21437/interspeech.2023-105.

[32] A. Sankar et al., “Towards building large scale datasets and state-of-the-art automatic speech translation systems for 14 Indian languages,” in Proc. 63rd Annu. Meeting Assoc. Comput. Linguistics (Vol. 1: Long Papers), 2025, doi: 10.18653/v1/2025.acl-long.1582.

[33] T. Javed et al., “IndicVoices: Towards building an inclusive multilingual speech dataset for Indian languages,” in Findings Assoc. Comput. Linguistics: ACL 2024, 2024, doi: 10.18653/v1/2024.findings-acl.639.

[34] T. Javed et al., “IndicSUPERB: A speech processing universal performance benchmark for Indian languages,” in Proc. AAAI Conf. Artif. Intell., 2023, doi: 10.1609/aaai.v37i11.26521.

[35] S. Kumar et al., “MADASR 2.0: Multi-lingual multi-dialect ASR challenge in 8 Indian languages,” in Proc. IEEE Autom. Speech Recognit. Understand. Workshop (ASRU), 2025, doi: 10.1109/ASRU65441.2025.11434729.

[36] A. Akhilesh et al., “Tamil speech recognition using XLSR Wav2Vec2.0 & CTC algorithm,” in Proc. 13th Int. Conf. Comput. Commun. Netw. Technol. (ICCCNT), 2022, doi: 10.1109/icccnt54827.2022.9984422.

[37] R. G. Rajakumari et al., “Multilingual automatic speech recognition for Indian languages-E2E framework,” in Proc. OITS Int. Conf. Inf. Technol. (OCIT), 2024, doi: 10.1109/ocit65031.2024.00033.

[38] T. Choudhary, V. Goyal, and A. Bansal, “WTASR: Wavelet transformer for automatic speech recognition of Indian languages,” Big Data Mining Anal., 2023, doi: 10.26599/bdma.2022.9020017.

[39] S. Balshankar and R. R. Deshmukh, “A critical insight into Marathi speech recognition and techniques,” in Proc. 2nd Int. Conf. Artif. Intell. Smart Energy (ICAIS), 2022, doi: 10.1109/icais53314.2022.9742925.

[40] N. Tasnia et al., “An overview of Bengali speech recognition: Methods, challenges, and future direction,” in Proc. IEEE 13th Annu. Comput. Commun. Workshop Conf. (CCWC), 2023, doi: 10.1109/ccwc57344.2023.10099382.

[41] C. H. Nga et al., “MLSS: Mandarin English code-switching speech recognition via mutual learning-based semi-supervised method,” IEEE Signal Process. Lett., 2025, doi: 10.1109/LSP.2025.3540953.

Share your valuable work from Social Media Buttons