Unsupervised Recurrent Neural Network with Parametric Bias Framework for Human Emotion Recognition with Multimodal Sensor Data Fusion

Li, Jie; Zhong, Junpei; Wang, Min

doi:10.18494/sam.2020.2552

Cited by 5 publications

(4 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to demonstrate the performance of emotion recognition of the proposed method, it is compared with the methods in Vivekanandam and Babu (2019), Jiang et al (2019), and (Li et al (2020) on the MOSI and MELD data sets. The recognition accuracy rates under different emotions are shown in Figure 9.…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

“…The recognition accuracy rates in both data sets are lower than 85%. Li et al (2020) proposed an emotion recognition framework based on RNN with parameter deviation. Emotion recognition is realized by fusing human body joint coordinates, angle and angular velocity to capture expressions.…”

Section: Comparison With Other Methodsmentioning

confidence: 99%

“…However, the application environment of this method is relatively special and lacks certain universality. Li et al (2020) proposed an emotion recognition framework based on RNN with parameter deviation. By fusing the coordinates, angles and angular velocity of human joints, expressions are captured for emotion recognition.…”

Section: Related Researchmentioning

confidence: 99%

See 2 more Smart Citations

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

show abstract

Section: Comparison With Other Methodsmentioning

confidence: 99%

Section: Comparison With Other Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…However, the environment for the application of this method is somehow specific and cannot be generalized. [15] raised an RNN-based emotion recognition framework with parameter deviation and recorded some expressions for recognizing emotions by combining the coordinates of the angles and the angular velocity of human joints.…”

Section: Related Studiesmentioning

confidence: 99%

Multimodal modeling of human emotions using sound, image and text fusion

Hosseini

Yamaghani

Arabani

2023

Preprint

View full text Add to dashboard Cite

Multimodal emotion recognition and analysis is considered a developing research field. Improving the multimodal fusion mechanism plays a key role in the more detailed recognition of the recognized emotion. The present study aimed to optimize the performance of the emotion recognition system and presented a model for multimodal emotion recognition from audio, text, and video data. First, the data were fused as a combination of video and audio, then as a combination of audio and text as binary, and finally the results were fused together. The final output included audio, text, and video data taking common features into account. Then, the convolutional neural network, as well as long-term and short-term memory (CNN-LSTM), were used to extract audio. Next, the Inception-Res Net-v2 network was applied for extracting the facial expression in the video. The output fused data were utilized by LSTM as the input of the softmax classifier to recognize the emotion of audio and video features fusion. In addition, the CNN-LSTM was combined in the form of a binary channel for learning audio emotion features. Meanwhile, a Bi-LSTM network was used to extract the text features and softmax was used for classifying the fused features. Finally, the generated results were fused together for the final classification, and the logistic regression model was used for fusion and classification. The results indicated that the recognition accuracy of the proposed method in the IEMOCAP data set was 82.9.

show abstract