A multimodal hierarchical approach to speech emotion recognition from audio and text

Singh, Prabhav; Srivastava, Ridam; Rana, K.P.S.; Kumar, Vineet

doi:10.1016/j.knosys.2021.107316

Cited by 65 publications

(40 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there has also been the use of Deep Learning models capable of processing hand-crafted features or a complete audio record. For example, Singh et al [39] proposed the use of hand-crafted features to feed deep neural networks. Prosody, spectral, and voice quality-based features were used to train a hierarchical DNN classifier, achieving an accuracy of 81.2% on the RAVDESS dataset.…”

Section: Speech Emotion Recognitionmentioning

confidence: 99%

“…Although vocal information is an essential modality for predicting emotions, the results of the emotion recognizer could improve by incorporating other modalities, as demonstrated by Singh et al [39], where incorporating textual features enhanced the results of the speech emotion recognizer. In our case, we include the information that facial expressions give for emotion recognition.…”

Section: Facial Emotion Recognitionmentioning

confidence: 99%

“…In this study, we used the RAVDESS dataset [18], mainly because it is a free of charge reference corpus for the scientific community for speech emotion recognition [39,67,68], but also because of its suitability for our experiments.…”

Section: The Dataset and Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

Jiménez

Griol

Callejas

et al. 2021

Sensors

View full text Add to dashboard Cite

Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a video-based task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.

show abstract

Section: Speech Emotion Recognitionmentioning

confidence: 99%

Section: Facial Emotion Recognitionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

Jiménez

Griol

Callejas

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…These classifiers are usually neural networks capable of processing these descriptors or the complete audio records. For example, Singh et al [33] suggested the use of prosody, spectral-information, and voice quality, to train a hierarchical DNN classifier, reaching an accuracy of 81.2% on RAVDESS. Pepino et al [34] combined eGeMAPS features with the embeddings extracted from an xlsr-Wav2Vec2.0 to train a CNN model.…”

Section: Speech Emotion Recognitionmentioning

confidence: 99%

“…Although voice is a crucial indicator of a subject's emotion, other modalities could enhance the SERs' performance, as demonstrated by Singh et al in [33], which incorporated textual features to supplement the speech emotion recognizer. In our scenario, we included the visual information of the facial expressions.…”

Section: Facial Emotion Recognitionmentioning

confidence: 99%

A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset

et al. 2021

View full text Add to dashboard Cite

Emotion recognition is attracting the attention of the research community due to its multiple applications in different fields, such as medicine or autonomous driving. In this paper, we proposed an automatic emotion recognizer system that consisted of a speech emotion recognizer (SER) and a facial emotion recognizer (FER). For the SER, we evaluated a pre-trained xlsr-Wav2Vec2.0 transformer using two transfer-learning techniques: embedding extraction and fine-tuning. The best accuracy results were achieved when we fine-tuned the whole model by appending a multilayer perceptron on top of it, confirming that the training was more robust when it did not start from scratch and the previous knowledge of the network was similar to the task to adapt. Regarding the facial emotion recognizer, we extracted the Action Units of the videos and compared the performance between employing static models against sequential models. Results showed that sequential models beat static models by a narrow difference. Error analysis reported that the visual systems could improve with a detector of high-emotional load frames, which opened a new line of research to discover new ways to learn from videos. Finally, combining these two modalities with a late fusion strategy, we achieved 86.70% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. Results demonstrated that these modalities carried relevant information to detect users’ emotional state and their combination allowed to improve the final system performance.

show abstract

Personality Recognition Models

Zataraín-Cabada

Lopez

2023

Multimodal Affective Computing

View full text Add to dashboard Cite

A multimodal hierarchical approach to speech emotion recognition from audio and text

Cited by 65 publications

References 40 publications

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset

Personality Recognition Models

Contact Info

Product

Resources

About