A Cascaded Burmese-to-Myeik Speech-to-Speech Translation System Using ASR, mBART-50 and MMS-VITS

Author

Ye Thu Aung, Thet Thet Zin

Abstract

 Speech-to-speech translation system translates speech from one language to another in speech form. However, developing this type of system for a low-resource language is challenging because only a limited amount of text and speech data is available. This paper proposes a Burmese to Myeik speech-to-speech translation system, which is based on Automatic Speech Recognition (ASR), Machine Translation (MT), and Text-to-Speech (TTS). First, CNN-BiLSTM-CTC model was used to recognize Burmese speech and transform it into Burmese text. Then, the obtained text was translated into Myeik language with the help of the fine-tuned mBART-50 model. As the recognized text may contain errors caused by the ASR model, the MT model was trained on both clean Burmese sentences and ASR-generated sentences. To obtain Myeik speech from the translated Myeik text, a pretrained Burmese MMS-VITS model was fine-tuned on the Myeik speech dataset. The results of the experiments were obtained as follows: The ASR model obtained a CER of 21.26% and SER of 27.29%. The mBART-50 model obtained a BLEU score of 13.53 on the clean Burmese-to-Myeik test set.  The robust MT model achieved a BLEU score of 13.09 when translating ASR-generated Burmese sentences. Finally, the experiments confirmed that the fine-tuned TTS model could generate intelligible Myeik speech. To conclude, the results of the experiments demonstrated that it is possible to develop a speech-to-speech translation system for the low-resource language based on a limited amount of training data.

Keywords

Burmese, Myeik, Speech-to-Speech Translation, Automatic Speech Recognition, Machine Translation, Text-to-Speech

Full Text:

Download Paper PDF

References

[1]    P. S. Krishna Paleti, B. Saluru, S. Vatturi, B. Yohoshiva and N. P. Challa, "Cascaded Speech-to-Speech Translation System Focusing on Low-Resource Indic     Languages," 2024 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON), New Delhi, India, 2024, pp. 1-6, doi: 10.1109/DELCON64804.2024.10867248.

[2]    Y. Jia et al., “Direct Speech-to-Speech Translation with a Sequence-to-Sequence Model,” in Proc. Interspeech, 2019, pp. 1123–1127.

[3]    A. N. Mon, W. P. Pa, and Y. K. Thu, “Building HMM-SGMM Continuous Automatic Speech Recognition on Myanmar Web News,” in Proceedings of the 15th International Conference on Computer Applications (ICCA 2017), Yangon, Myanmar, pp. 446–453, 2017.

[4]    Chit, K.M.M., Lin, L.L. (2021). Exploring CTC Based End-To-End Techniques for Myanmar Speech Recognition. In: Vasant, P., Zelinka, I., Weber, GW. (eds) Intelligent Computing and Optimization. ICO 2020. Advances in Intelligent Systems and Computing, vol 1324. Springer, Cham. https://doi.org/10.1007/978-3-030-68154-8_87

[5]   T. M. Oo, Y. K. Thu, K. M. Soe, and T. Supnithi, “Statistical Machine Translation between Myanmar and Myeik,” in Proceedings of the 12th International Conference on Future Computer and Communication (ICFCC 2020), Yangon, Myanmar, Feb. 26–28, 2020, pp. 36–45.  

[6]   T. M. Oo, Y. K. Thu, K. M. Soe and T. Supnithi, "Neural Machine Translation between Myanmar (Burmese) and Dawei (Tavoyan)," 2020 IEEE Conference on Computer Applications (ICCA), Yangon, Myanmar, 2020, pp. 1-7, doi: 10.1109/ICCA49400.2020.9022813.

[7]    Y. Liu et al., “Multilingual Denoising Pre-training for Neural Machine Translation,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, 2020.

[8]    Y. Tang, C. Tran, X. Li, P.-J. Chen, N. Goyal, V. Chaudhary, J. Gu, and A. Fan, “Multilingual Translation with Extensible Multilingual Pretraining and Finetuning,” arXiv preprint arXiv:2008.00401, 2020.

[9]    V. Pratap et al., “Scaling Speech Technology to 1,000+ Languages,” Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024.

[10] J. Shen et al., “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779–4783, 2018.

[11]   J. Kim, J. Kong, and J. Son, “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech,” in Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 5530–5540, 2021.

[12]   Jia, Ye et al. “Translatotron 2: High-quality direct speech-to-speech translation with voice preservation.” International Conference on Machine Learning (2021).

[13]    L. Barrault et al., “SeamlessM4T—Massively Multilingual & Multimodal Machine Translation,” arXiv preprint arXiv:2308.11596, 2023.

[14]   Y. M. Oo et al., “Burmese Speech Corpus, Finite-State Text Normalization and Pronunciation Grammars with an Application to Text-to-Speech,” in Proceedings of the 12th Language Resources and Evaluation Conference (LREC), Marseille, France, 2020, pp. 6328–6339.

[15]    Chuu Htet Naing, “Myanmar Speech Dataset for ASR,”. Available:  https://huggingface.co/datasets/chuuhtetnaing/myanmar-speech-dataset-for-asr

[16]    A. Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems, vol. 30, 2017.

[17]    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th   Annual Meeting of the Association for Computational Linguistics, pp. 311–318, 2002.

[18]    K. Heafield, “KenLM: Faster and Smaller Language Model Queries,” in Proceedings of the Sixth Workshop on Statistical Machine Translation, pp. 187–197, 2011.

[19] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,” Proceedings of the 23rd International Conference on Machine Learning (ICML), 2006, pp. 369–376.

 

Share your valuable work from Social Media Buttons