Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals

Choi, Dong Yoon; Kim, Deok‐Hwan; Song, Byung Cheol

doi:10.1109/access.2020.3036877

Cited by 16 publications

(7 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fusing multimodal features can achieve a better emotion recognition effect. For example, Choi et al [20] proposed a multimodal fusion network integrating video modality and EEG modality. To calculate the attention weights of facial video features and corresponding EEG features, they described a bilinear pooled multimodal attention network based on low-rank decomposition.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Wang

Zhang

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Emotion recognition has attracted a lot of attention in recent years. It is widely used in health care, teaching, human-computer interaction, and other fields. Various human emotional features are often used to recognize different emotions. Currently, there is more and more research on multimodal emotion recognition based on the fusion of multiple features. This paper proposes a deep learning model of multimodal emotion recognition based on the fusion of electroencephalogram (EEG) signals and facial expressions to achieve an excellent classification effect. First, the pre-trained convolution neural network (CNN) is used to extract the facial features from the facial expressions. Next, the attention mechanism is introduced to extract more critical facial frame features. Then, we apply CNNs to extract spatial features from original EEG signals, which use a local convolution kernel and a global convolution kernel to learn the features of left and right hemispheres channels and all EEG channels. After feature-level fusion, the fusion features of the facial expression features and EEG features are fed into the classifier for emotion recognition. This paper conducted experiments on the DEAP and MAHNOB-HCI datasets to evaluate the performance of the proposed model. The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset. The experimental results show that the proposed model can effectively carry out emotion recognition.

show abstract

Section: Related Workmentioning

confidence: 99%

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Wang

Zhang

et al. 2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Most of these works use images, audio and/or text as inputs [2-5, 20, 21]. In a few cases, physiological signals have been used to improve recognition from image, audio and text [22][23][24]. A few authors have described the use of multiple physiological signal modalities [6][7][8].…”

Section: Related Workmentioning

confidence: 99%

Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals

Vazquez-Rodriguez

Lefebvre

Cumin

et al. 2022

2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

In this paper, we address the problem of multimodal emotion recognition from multiple physiological signals. We demonstrate that a Transformer-based approach is suitable for this task. In addition, we present how such models may be pretrained in a multimodal scenario to improve emotion recognition performances. We evaluate the benefits of using multimodal inputs and pre-training with our approach on a state-ofthe-art dataset.

show abstract

“…DL is exploited as a extractor of feature on feature map and later feature extracted is combined for the classification method to identify dissimilar types of emotion. Choi et al [13] introduced a multi-modal fusion network that incorporates EEG and video modality networks. To compute attention weight of the corresponding EEG and facial video features, a multimodal attention network using bi-linear pooling related to low-rank decomposition, is developed.…”

Section: Related Workmentioning

confidence: 99%

Enhanced Bird Swarm Algorithm with Deep Learning based Electroencephalography Signal Analysis for Emotion Recognition

Al-Farouni

2022

Journal of Smart Internet of Things

View full text Add to dashboard Cite

Bioelectric signals comprise a massive count of data, and researchers in various domains containing cognitive neuroscience, psychiatry, and so on. Emotion is a vital part of regular human communication. The emotional conditions and dynamics of brain are connected by electroencephalography (EEG) signal which is utilized by Brain-Computer Interface (BCI), for providing optimum human-machine interaction. EEG-based emotion detection was extremely utilized in military, human-computer interactions, medicinal analysis, and other domains. Identifying emotions utilizing biological brain signals need accurate and effectual signal processing and extracting features approaches. But, one of the essential problems facing the emotion detection method, utilizing EEG signal is the detection accuracy. In this aspect, this study develops an Enhanced Bird Swarm Algorithm with Deep Learning based Electroencephalography Signal Analysis for Emotion Recognition (EBSADL-ESEG) technique. The ultimate aim of the EBSADL-ESEG technique lies in the recognition of emotions using the EEG signals accurately. To perform this, the EBSADL-ESEG technique initially extracts the statistical features from the EEG signals. In addition, the EBSA technique is employed for optimal feature selection process. Moreover, the gated recurrent unit (GRU) with root mean square propagation (RMSProp) optimizer is utilized for classifying distinct emotions (arousal, valence, and liking). The experimental analysis of the EBSADL-ESEG model is tested on DEAP dataset and the outcomes are investigated under diverse measures. The comprehensive comparison study revealed better outcomes of the EBSADL-ESEG model over other DL models.

show abstract

Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals

Cited by 16 publications

References 50 publications

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals

Enhanced Bird Swarm Algorithm with Deep Learning based Electroencephalography Signal Analysis for Emotion Recognition

Contact Info

Product

Resources

About