Multimodal Speech Emotion Recognition Using Audio and Text

Yoon, Seunghyun; Byun, Seokhyun; Jung, Kyomin

doi:10.1109/slt.2018.8639583

Cited by 223 publications

(143 citation statements)

References 27 publications

Supporting

Mentioning

135

Contrasting

Unclassified

Order By: Relevance

“…In [19], emotional keywords are exploited to effectively identify the classes. Recently in [9,10,20], a long shortterm memory (LSTM) based network has been explored to encode the information of both modalities. Furthermore, there have been some attempts to fuse the modalities using the inter-attention mechanism [11,12].…”

Section: Recent Workmentioning

confidence: 99%

“…This recurrent encoder will be used in the same manner to the audio, text and video modality, independently. One slight difference exists in the audio modality case such that we add an additional prosodic feature vector to the output of audio recurrent encoder following a previous research [9]. For the video data, we obtain a fixed dimensional representation of each frame from a pretrained ResNet.…”

Section: Recurrent Encodermentioning

confidence: 99%

“…Figure 1 shows the architecture of the proposed AMH model. Previous research used multi-modal information independently using a neural network model by fusion information over each modality [9,20]. Recently, researchers also investigated an interattention mechanism over the modality [11,12].…”

Section: Proposed Attentive Modality Hopping Mechanismmentioning

confidence: 99%

“…A total of 10 unique speakers participated in this work. Following the previous research [9,10,12], we assign single categorical emotion to the utterance with the majority of annotators agreed on the emotion labels. The final dataset contains 7,487 utterances in total (1,103 angry, 1,041 excite, 595 happy, 1,084 sad, 1,849 frustrated, 107 surprise and 1,708 neutral).…”

Section: Dataset and Experimental Setupmentioning

confidence: 99%

See 3 more Smart Citations

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Yoon

Dey

Lee

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

In this work, we explore the impact of visual modality in addition to speech and text for improving the accuracy of the emotion detection system. The traditional approaches tackle this task by fusing the knowledge from the various modalities independently for performing emotion classification. In contrast to these approaches, we tackle the problem by introducing an attention mechanism to combine the information. In this regard, we first apply a neural network to obtain hidden representations of the modalities. Then, the attention mechanism is defined to select and aggregate important parts of the video data by conditioning on the audio and text data. Furthermore, the attention mechanism is again applied to attend important parts of the speech and textual data, by considering other modality. Experiments are performed on the standard IEMOCAP dataset using all three modalities (audio, text, and video). The achieved results show a significant improvement of 3.65% in terms of weighted accuracy compared to the baseline system.

show abstract

Section: Recent Workmentioning

confidence: 99%

Section: Recurrent Encodermentioning

confidence: 99%

Section: Proposed Attentive Modality Hopping Mechanismmentioning

confidence: 99%

Section: Dataset and Experimental Setupmentioning

confidence: 99%

See 2 more Smart Citations

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Yoon

Dey

Lee

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In emotion recognition, many studies extracted features from audio, visual, or textual domains and then fuse them either in the feature levels or decision levels [18,19,20]. To leverage information from speech signals and text sequences, previous study [21] used neural networks to model two sequences separately and use direct concatenation of two modalities for emotion classification. In [22], a tensor fusion network was proposed to fuse features from different modalities and learn intra-modality and inter-modality dynamics.…”

Section: Related Workmentioning

confidence: 99%

Learning Alignment for Multimodal Emotion Recognition from Speech

Xu¹,

Zhang²,

Han³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

Speech emotion recognition is a challenging problem because human convey emotions in subtle and complex ways. For emotion recognition on human speech, one can either extract emotion related features from audio signals or employ speech recognition techniques to generate text from speech and then apply natural language processing to analyze the sentiment. Further, emotion recognition will be beneficial from using audio-textual multimodal information, it is not trivial to build a system to learn from multimodality. One can build models for two input sources separately and combine them in a decision level, but this method ignores the interaction between speech and text in the temporal domain. In this paper, we propose to use an attention mechanism to learn the alignment between speech frames and text words, aiming to produce more accurate multimodal feature representations. The aligned multimodal features are fed into a sequential model for emotion recognition. We evaluate the approach on the IEMOCAP dataset and the experimental results show the proposed approach achieves the state-of-the-art performance on the dataset.

show abstract

A learning framework of modified deep recurrent neural network for classification and recognition of voice mood

Agarwal

Gupta

2022

Adaptive Control & Signal

View full text Add to dashboard Cite

Recognition of human emotions is a basic requirement in many real-time applications. Detection of exact emotions through voice provides relevant information for various purposes. Several computational methods have been employedfor the clear analysis of human emotions. Most of the previous approaches face complexities due to certain drawbacks like degraded signal quality, a requirement of high storage space, increased computational complexity, and deteriorated outcomes of classification accuracy. The proposed work was implemented to gather the accurate classification result of embedded emotions and minimize the computational complexities of MDDTRNN (modified deep duck and traveler recurrent neural network). The proposed work includes four steps: preprocessing, feature extraction, feature selection, and classification. In feature extraction, the spectral and frequency features are extracted using the adopting boosted MFCC (Mel frequency cepstral coefficients) method to improve training speed. In feature selection, the best features are selected using an algorithm of AAVOA (adaptive African vulture optimization algorithm). To provide optimal emotion results, the classification step is undertaken by the MDDTRNN technique. The proposed work shows better classification outcomes of emotions when compared to the existing approaches by holding the accuracy of (95.86%), precision as (93.79%), specificity as (94.28%), sensitivity as (92.89%) and the error rate is attained to be 5.266 in terms of IEMOCAP dataset. The accuracy result (96.27%), precision (94.83%), specificity (93.16%), sensitivity (94%) and the error rate is achieved to be 4.982 in terms of the EMODB dataset.

show abstract

Multimodal Speech Emotion Recognition Using Audio and Text

Cited by 223 publications

References 27 publications

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Attentive Modality Hopping Mechanism for Speech Emotion Recognition

Learning Alignment for Multimodal Emotion Recognition from Speech

A learning framework of modified deep recurrent neural network for classification and recognition of voice mood

Contact Info

Product

Resources

About