Learning-Based Confidence Estimation for Multi-modal Classifier Fusion

Nadeem, Uzair; Bennamoun, Mohammed; Sohel, Ferdous; Togneri, Roberto

doi:10.1007/978-3-030-36711-4_26

Cited by 2 publications

(1 citation statement)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their work included a Hierarchical Memory Network (HMN) with a bidirectional GRU (BiGRU) as the utterance reader and a BiGRU fusion layer for the interaction between historical utterances For memory summarizing, they propose an Attention GRU (AGRU) to utilize the attention weights to update the internal state of GRU. Besides, Nadeem et al [25] intended a novel confidence estimation method for predictions from a multi-class emotional classifier. The predicted confidence values by the proposed system are used to improve the accuracy of multi-modal emotion.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Approach of Speech Emotion Recognition Using Multi-Level Multi-Head Fusion Attention-Based Recurrent Neural Network

et al. 2020

View full text Add to dashboard Cite

Speech emotion recognition is a challenging but important task in human computer interaction (HCI). As technology and understanding of emotion are progressing, it is necessary to design robust and reliable emotion recognition systems that are suitable for real-world applications both to enhance analytical abilities supporting human decision making and to design human-machine interfaces (HMI) that assist efficient communication. This paper presents a multimodal approach for speech emotion recognition based on Multi-Level Multi-Head Fusion Attention mechanism and recurrent neural network (RNN). The proposed structure has inputs of two modalities: audio and text. For audio features, we determine the mel-frequency cepstrum (MFCC) from raw signals using the OpenSMILE toolbox. Further, we use pre-trained model of bidirectional encoder representations from transformers (BERT) for embedding text information. These features are fed parallelly into the self-attention mechanism base RNNs to exploit the context for each timestamp, then we fuse all representatives using multi-head attention technique to predict emotional states. Our experimental results on the three databases: Interactive Emotional Motion Capture (IEMOCAP), Multimodal EmotionLines Dataset (MELD), and CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI), reveal that the combination of the two modalities achieves better performance than using single models. Quantitative and qualitative evaluations on all introduced datasets demonstrate that the proposed algorithm performs favorably against state-of-the-art methods. INDEX TERMS Speech emotion recognition, multi-level multi-head fusion attention, RNN, audio features, textual features.

show abstract

Section: Related Workmentioning

confidence: 99%