Semi-supervised learning for facial expression-based emotion recognition in the continuous domain

Choi, Dong Yoon; Song, Byung Cheol

doi:10.1007/s11042-020-09412-5

Cited by 13 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The feature extraction module is the basis of the multimodal emotion recognition method which aims to receive the input for information processing and transforms it into the features which can be used by the model [32]. Various schemes are considered in the feature extraction module for extracting features.…”

Section: Fig 2 Multimodal Emotion Recognition Methods Based On Deep L...mentioning

confidence: 99%

Multimodal modeling of human emotions using sound, image and text fusion

Hosseini

Yamaghani

Arabani

2023

Preprint

View full text Add to dashboard Cite

Multimodal emotion recognition and analysis is considered a developing research field. Improving the multimodal fusion mechanism plays a key role in the more detailed recognition of the recognized emotion. The present study aimed to optimize the performance of the emotion recognition system and presented a model for multimodal emotion recognition from audio, text, and video data. First, the data were fused as a combination of video and audio, then as a combination of audio and text as binary, and finally the results were fused together. The final output included audio, text, and video data taking common features into account. Then, the convolutional neural network, as well as long-term and short-term memory (CNN-LSTM), were used to extract audio. Next, the Inception-Res Net-v2 network was applied for extracting the facial expression in the video. The output fused data were utilized by LSTM as the input of the softmax classifier to recognize the emotion of audio and video features fusion. In addition, the CNN-LSTM was combined in the form of a binary channel for learning audio emotion features. Meanwhile, a Bi-LSTM network was used to extract the text features and softmax was used for classifying the fused features. Finally, the generated results were fused together for the final classification, and the logistic regression model was used for fusion and classification. The results indicated that the recognition accuracy of the proposed method in the IEMOCAP data set was 82.9.

show abstract

Section: Fig 2 Multimodal Emotion Recognition Methods Based On Deep L...mentioning

confidence: 99%

Multimodal modeling of human emotions using sound, image and text fusion

Hosseini

Yamaghani

Arabani

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The feature extraction module is the basis of the entire multi-modal emotion recognition model. It is mainly used to process the input modal information and turn it into features that can be used by the model (Choi and Song, 2020). In the feature extraction module, different feature extraction schemes are set for different single modes.…”

Section: Proposed Multi-modal Emotion Recognition Methods For Speech Expression Overall Model Architecturementioning

confidence: 99%

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Liu

Wang

et al. 2021

Front. Neurorobot.

View full text Add to dashboard Cite

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

show abstract

“…In emotion recognition tasks, because of the continuity of facial expressions and EEG signals, emotional expressions are highly correlated in time series. However, single-point facial expression pictures and EEG signal data are often in the process of facial expression changes, which are prone to misjudgment [29,30]. Therefore, for sequence emotion recognition tasks, LSTM's processing of sequences is similar to the processing method of the human brain on emotion recognition tasks, and the algorithm has the advantage of natural adaptability.…”

Section: Related Workmentioning

confidence: 99%

Expression-EEG Bimodal Fusion Emotion Recognition Method Based on Deep Learning

Zhang

Shi

et al. 2021

Computational and Mathematical Methods in Medicine

View full text Add to dashboard Cite

As one of the key issues in the field of emotional computing, emotion recognition has rich application scenarios and important research value. However, the single biometric recognition in the actual scene has the problem of low accuracy of emotion recognition classification due to its own limitations. In response to this problem, this paper combines deep neural networks to propose a deep learning-based expression-EEG bimodal fusion emotion recognition method. This method is based on the improved VGG-FACE network model to realize the rapid extraction of facial expression features and shorten the training time of the network model. The wavelet soft threshold algorithm is used to remove artifacts from EEG signals to extract high-quality EEG signal features. Then, based on the long- and short-term memory network models and the decision fusion method, the model is built and trained using the signal feature data extracted under the expression-EEG bimodality to realize the final bimodal fusion emotion classification and identification research. Finally, the proposed method is verified based on the MAHNOB-HCI data set. Experimental results show that the proposed model can achieve a high recognition accuracy of 0.89, which can increase the accuracy of 8.51% compared with the traditional LSTM model. In terms of the running time of the identification method, the proposed method can effectively be shortened by about 20 s compared with the traditional method.

show abstract

Semi-supervised learning for facial expression-based emotion recognition in the continuous domain

Cited by 13 publications

References 25 publications

Multimodal modeling of human emotions using sound, image and text fusion

Multimodal modeling of human emotions using sound, image and text fusion

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Expression-EEG Bimodal Fusion Emotion Recognition Method Based on Deep Learning

Contact Info

Product

Resources

About