Emotinal Recognition Using Multimodel LLM

Author

Mrs. Gunasundhari M , Kavenilaa B, Jenova J , Kaviya A, Jaihindh V

Abstract

 Emotions are an important part of human communication, but identifying them accurately from a single source can be difficult. The proposed Emotional Recognition Using Multimodal LLM .system combines text, speech, and facial expressions to understand a person’s emotional state Text is analyzed for emotional meaning, speech is examined using features such as tone and pitch, and facial expressions are processed to identify visible emotions. These individual results are combined using a Large Language Model (LLM) to generate a final emotion prediction. By considering multiple forms of communication, the system aims to provide a more reliable and context-aware understanding of human emotions. The proposed approach can support ,applications in human-computer interaction, virtual assistants education, and other intelligent systems

Keywords

ultimodal Emotion Recognition, Large Language Model, Facial Expression, Speech Emotion, Text Emotion, Artificial Intelligence.

Full Text:

Download Paper PDF

References

1.A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face — H. Lian,
C. Lu, S. Li, Y. Zhao, C. Tang & Y. Zong (2023). Reviews deep learning methods for recognizing emotions from speech, text, and facial expressions. (MDPI)
2.Deep Learning-Based Multimodal Emotion Recognition from Audio, Visual, and Text Modalities — S. Zhang, Y. Yang, C. Chen, X. Zhang, Q. Leng & X. Zhao (2024). Reviews recent approaches for combining audio, visual, and text information for emotion recognition. (ScienceDirect)
3.MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations — S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria & R. Mihalcea (2019). Introduces a multimodal dataset containing text, audio, and visual information for emotion recognition. (ACL Anthology)
4.The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) — S. R. Livingstone &
F. A. Russo (2018). Presents a dataset containing facial and vocal emotional expressions for emotion-related research. (PubMed Central (PMC))
5.Deep Learning Based Multimodal Emotion Recognition Using Model-Level Fusion of Audio-Visual
Modalities — A. I. Middya, B. Nag & S. Roy (2022). Explores the combination of audio and visual features for multimodal emotion recognition. (ScienceDirect)
6.DeepFace: Closing the Gap to Human-Level Performance in Face Verification — Y. Taigman, M. Yang,
M. Ranzato & L. Wolf (2014). Presents a deep learning approach for extracting meaningful facial representations from images. (Open Access CVF)
7.BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — J. Devlin, M.-
W. Chang, K. Lee & K. Toutanova (2019). Introduces BERT, a transformer-based model widely used for understanding and classifying text.

Share your valuable work from Social Media Buttons