Spatial-Temporal Feature Fusion Neural Network for EEG-Based Emotion Recognition

Wang, Zhe; Wang, Yongxiong; Zhang, Jiapeng; Hu, Chuanfei; Yin, Zhong; Song, Yu

doi:10.1109/tim.2022.3165280

Cited by 36 publications

(9 citation statements)

References 51 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Zhong et al [17] introduce a regularized graph neural network for EEG-based emotion recognition, incorporating the biological topology of the brain. Wang et al [18] propose a hybrid spatial-temporal feature fusion neural network that simultaneously leverages the extensive emotional information embedded in EEG signals across both temporal and spatial dimensions, aiming at more effectively encapsulating and representing subjects' emotional states. Ye et al [19] present the hierarchical dynamic graph convolutional network, designed to extract dynamic multilayer spatial information interlinking EEG signal channels, thereby enhancing the accuracy of emotion recognition.…”

Section: Eeg-based Multimodal Emotion Recognition Methodsmentioning

confidence: 99%

Cross-modal credibility modelling for EEG-based multimodal emotion recognition

Zhang,

Liu,

Wang

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

Objective. The study of emotion recognition through electroencephalography (EEG) has garnered significant attention recently. Integrating EEG with other peripheral physiological signals may greatly enhance performance in emotion recognition. Nonetheless, existing approaches still suffer from two predominant challenges: modality heterogeneity, stemming from the diverse mechanisms across modalities, and fusion credibility, which arises when one or multiple modalities fail to provide highly credible signals. Approach. In this paper, we introduce a novel multimodal physiological signal fusion model that incorporates both intra-inter modality reconstruction and sequential pattern consistency, thereby ensuring a computable and credible EEG-based multimodal emotion recognition. For the modality heterogeneity issue, we first implement a local self-attention transformer to obtain intra-modal features for each respective modality. Subsequently, we devise a pairwise cross-attention transformer to reveal the inter-modal correlations among different modalities, thereby rendering different modalities compatible and diminishing the heterogeneity concern. For the fusion credibility issue, we introduce the concept of sequential pattern consistency to measure whether different modalities evolve in a consistent way. Specifically, we propose to measure the varying trends of different modalities, and compute the inter-modality consistency scores to ascertain fusion credibility. Main results. We conduct extensive experiments on two benchmarked datasets (DEAP and MAHNOB-HCI) with the subject-dependent paradigm. For the DEAP dataset, our method improves the accuracy by 4.58%, and the F1 score by 0.63%, compared to the state-of-the-art baseline. Similarly, for the MAHNOB-HCI dataset, our method improves the accuracy by 3.97%, and the F1 score by 4.21%. In addition, we gain much insight into the proposed framework through significance test, ablation experiments, confusion matrices and hyperparameter analysis. Consequently, we demonstrate the effectiveness of the proposed credibility modelling through statistical analysis and carefully designed experiments. Significance. All experimental results demonstrate the effectiveness of our proposed architecture and indicate that credibility modelling is essential for multimodal emotion recognition.

show abstract

Section: Eeg-based Multimodal Emotion Recognition Methodsmentioning

confidence: 99%

Cross-modal credibility modelling for EEG-based multimodal emotion recognition

Zhang,

Liu,

Wang

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

show abstract

“…Capturing spatial correlation among electrodes could provide discriminative information to recognize emotional states. Wang et al (2022b) transformed EEG signals into 2D topographic maps and adopted a convolutional neural network (CNN) to learn the spatial dependencies. Although this method enables EEG spatial learning, the high dimensional topographic maps obtained by the interpolation method could potentially cause feature redundancy.…”

Section: Related Work 21 Spatial Learning Of Eeg Signals For Emotion ...mentioning

confidence: 99%

Cerebral asymmetry representation learning-based deep subdomain adaptation network for electroencephalogram-based emotion recognition

Wang,

Wan

et al. 2024

Physiol. Meas.

View full text Add to dashboard Cite

Objective. Extracting discriminative spatial information from multiple electrodes is a crucial and challenging problem for Electroencephalogram (EEG)-based emotion recognition. Additionally, the domain shift caused by the individual differences would degrade the performance of cross-subject EEG classification. Approach. To deal with the problems, we propose the Cerebral Asymmetry Representation Learning based Deep Subdomain Adaptation Network (CARL-DSAN) to enhance the cross-subject EEG-based emotion recognition. Specifically, the CARL module is inspired by the neuroscience findings that asymmetrical activations of left and right brain hemispheres occur during cognitive and affective processes. In the CARL module, we introduce a novel two-step strategy for extracting discriminative features through intra-hemisphere spatial learning and asymmetry representation learning. Moreover, the transformer encoders within the CARL module could emphasize the contributive electrodes and electrode pairs. Subsequently, the DSAN module, known for its superior performance over global domain adaption, is adopted to mitigate domain shift and further improve the cross-subject performance by aligning relevant subdomains which share the same class samples. Main Results. To validate the effectiveness of the CARL-DSAN, we conduct the subject-independent experiments on the DEAP database, achieving the accuracies of 68.67% and 67.11% for arousal and valence classification respectively, and corresponding accuracies of 67.70% and 67.18% on the MAHNOB-HCI database. Significance. The results demonstrate that CARL-DSAN could achieve outstanding cross-subject performance in both arousal and valence classification.

show abstract

“…The transformer-based model emerges as a power-ful approach, capable of learning discriminative spatial information extending from the electrode level to the brain-region level, in the pursuit of improving the ability to capture EEG spatial dependencies and enhance the accuracy of emotion recognition. [22,23].…”

Section: Exploring Past Work: a Brief Literature Reviewmentioning

confidence: 99%

Integrated Modeling for In-Depth EEG Based Emotional State Analysis via Convolutional-Transformer Fusion

Ali,

Kang,

Himral

2023

Preprint

View full text Add to dashboard Cite

This study introduces a hybrid model designed for the predic- tion of emotional states derived from electroencephalogram (EEG) data, employing an amalgamation of convolutional and trans- former layers. The architectural framework of the model is metic- ulously structured to facilitate concurrent assimilation of local pat- terns and long-range dependencies inherent in EEG data, thereby augmenting its discernment of nuanced emotional experiences. The investigation presented herein undertakes a comprehensive explo- ration of the fusion technique, with a primary focus on dis- cerning three elemental emotional dimensions: Arousal, Valence, and Dominance, in addition to their concurrent combinations. The research methodology encompasses an in-depth evaluation of the model’s performance across these diverse emotional states, encompassing the intricate task of simultaneous Valence-Arousal (VA) prediction. Furthermore, the study extends its purview to encompass the intricate Valence-Arousal-Dominance (VAD) space, thereby providing a thorough analysis of the model’s efficacy. To articulate the model’s discriminative efficacy, this study meticu- lously presents the detailed F1 scores corresponding to each emotional state classification: Arousal (96.8), Valence (97.3), Valence-Arousal (VA) simultaneously (95.6), and Valence-Arousal-Dominance simultaneously (94.9). These scores serve as a testament to the model’s robust per- formance across diverse emotional dimensions. Importantly, to fortify the credibility of our findings, rigorous experimentation has been con- ducted on the DEAP dataset, unveiling noteworthy results even in scenarios involving simultaneous recognition of multiple emotional states.

show abstract

Spatial-Temporal Feature Fusion Neural Network for EEG-Based Emotion Recognition

Cited by 36 publications

References 51 publications

Cross-modal credibility modelling for EEG-based multimodal emotion recognition

Cross-modal credibility modelling for EEG-based multimodal emotion recognition

Cerebral asymmetry representation learning-based deep subdomain adaptation network for electroencephalogram-based emotion recognition

Integrated Modeling for In-Depth EEG Based Emotional State Analysis via Convolutional-Transformer Fusion

Contact Info

Product

Resources

About