Emotion recognition in EEG signals using deep learning methods: A review

Jafari, Mahboobeh; Shoeibi, Afshin; Khodatars, Marjane; Bagherzadeh, Sara; Shalbaf, Ahmad; García, David López; Gorriz, Juan M.; Acharya, U. Rajendra

doi:10.1016/j.compbiomed.2023.107450

Cited by 40 publications

(11 citation statements)

References 244 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In studies that aim to improve model interpretability, this often requires that the individual frequency bands of the EEG signal be extracted separately or that the EEG signal be time-frequency transformed before feeding into the model ( Maheshwari et al, 2021 ). In clinical settings, achieving high accuracy and interpretability is of utmost importance ( Ribeiro et al, 2016 ; Jafari et al, 2023 ). Black-box CNN models make it difficult to test decisions and fail to explain if their intrinsic representations correspond with clinical features, leading to doctors’ skepticism and hindering translational applications.…”

Section: Discussionmentioning

confidence: 99%

“…The Explainable AI (XAI) in EEG emotion recognition will be a critical area of future research. Not only will it help researchers validate existing medical knowledge or discover new ones, as Mayor Torres et al (2023) using the explainable deep learning algorithm SincNet to identify high-alpha and beta suppression in EEG signals of individuals with autism spectrum disorders, but it will also increase physicians’ confidence in using deep learning for diagnosis ( Jafari et al, 2023 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ERTNet: an interpretable transformer-based framework for EEG emotion recognition

Liu,

Chao,

et al. 2024

Front. Neurosci.

View full text Add to dashboard Cite

BackgroundEmotion recognition using EEG signals enables clinicians to assess patients’ emotional states with precision and immediacy. However, the complexity of EEG signal data poses challenges for traditional recognition methods. Deep learning techniques effectively capture the nuanced emotional cues within these signals by leveraging extensive data. Nonetheless, most deep learning techniques lack interpretability while maintaining accuracy.MethodsWe developed an interpretable end-to-end EEG emotion recognition framework rooted in the hybrid CNN and transformer architecture. Specifically, temporal convolution isolates salient information from EEG signals while filtering out potential high-frequency noise. Spatial convolution discerns the topological connections between channels. Subsequently, the transformer module processes the feature maps to integrate high-level spatiotemporal features, enabling the identification of the prevailing emotional state.ResultsExperiments’ results demonstrated that our model excels in diverse emotion classification, achieving an accuracy of 74.23% ± 2.59% on the dimensional model (DEAP) and 67.17% ± 1.70% on the discrete model (SEED-V). These results surpass the performances of both CNN and LSTM-based counterparts. Through interpretive analysis, we ascertained that the beta and gamma bands in the EEG signals exert the most significant impact on emotion recognition performance. Notably, our model can independently tailor a Gaussian-like convolution kernel, effectively filtering high-frequency noise from the input EEG data.DiscussionGiven its robust performance and interpretative capabilities, our proposed framework is a promising tool for EEG-driven emotion brain-computer interface.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

ERTNet: an interpretable transformer-based framework for EEG emotion recognition

Liu,

Chao,

et al. 2024

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Incorporating various noninvasive methods such as EEG [ 19 , 20 ], fNIRS [ 21 ], eye-tracking [ 22 , 23 ], and VR/AR integrations [ 24 , 25 ], BCIs promise wide-ranging applications. These include facilitating communication for those with disabilities [ 19 , 26 ] and enriching immersive gaming and virtual reality experiences [ 27 ], as well as roles in disease diagnosis [ 28 , 29 ] and mental state monitoring [ 30 , 31 ]. BCI technology’s expanding abilities herald a new frontier in healthcare, entertainment, and education, marking a significant leap in human–computer interaction.…”

Section: Related Workmentioning

confidence: 99%

EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN

Arif,

Wang,

Yin

et al. 2024

Sensors

View full text Add to dashboard Cite

Analysis of brain signals is essential to the study of mental states and various neurological conditions. The two most prevalent noninvasive signals for measuring brain activities are electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, characterized by its higher sampling frequency, captures more temporal features, while fNIRS, with a greater number of channels, provides richer spatial information. Although a few previous studies have explored the use of multimodal deep-learning models to analyze brain activity for both EEG and fNIRS, subject-independent training–testing split analysis remains underexplored. The results of the subject-independent setting directly show the model’s ability on unseen subjects, which is crucial for real-world applications. In this paper, we introduce EF-Net, a new CNN-based multimodal deep-learning model. We evaluate EF-Net on an EEG-fNIRS word generation (WG) dataset on the mental state recognition task, primarily focusing on the subject-independent setting. For completeness, we report results in the subject-dependent and subject-semidependent settings as well. We compare our model with five baseline approaches, including three traditional machine learning methods and two deep learning methods. EF-Net demonstrates superior performance in both accuracy and F1 score, surpassing these baselines. Our model achieves F1 scores of 99.36%, 98.31%, and 65.05% in the subject-dependent, subject-semidependent, and subject-independent settings, respectively, surpassing the best baseline F1 scores by 1.83%, 4.34%, and 2.13% These results highlight EF-Net’s capability to effectively learn and interpret mental states and brain activity across different and unseen subjects.

show abstract

“…Current research predominantly integrates physiological data like EEG and eye movement recordings with behavioral data to formulate advanced emotion recognition systems. Such multimodal approaches have been shown to yield accurate insights into user emotions ( Khosla, Khandnor & Chand, 2020 ; Lim, Mountstephens & Teo, 2020 ; Jafari et al, 2023 ).…”

Section: Introductionmentioning

confidence: 99%

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Guo,

Yang,

Lin

et al. 2024

PeerJ Computer Science

View full text Add to dashboard Cite

Emotional recognition is a pivotal research domain in computer and cognitive science. Recent advancements have led to various emotion recognition methods, leveraging data from diverse sources like speech, facial expressions, electroencephalogram (EEG), electrocardiogram, and eye tracking (ET). This article introduces a novel emotion recognition framework, primarily targeting the analysis of users’ psychological reactions and stimuli. It is important to note that the stimuli eliciting emotional responses are as critical as the responses themselves. Hence, our approach synergizes stimulus data with physical and physiological signals, pioneering a multimodal method for emotional cognition. Our proposed framework unites stimulus source data with physiological signals, aiming to enhance the accuracy and robustness of emotion recognition through data integration. We initiated an emotional cognition experiment to gather EEG and ET data alongside recording emotional responses. Building on this, we developed the Emotion-Multimodal Fusion Neural Network (E-MFNN), optimized for multimodal data fusion to process both stimulus and physiological data. We conducted extensive comparisons between our framework’s outcomes and those from existing models, also assessing various algorithmic approaches within our framework. This comparison underscores our framework’s efficacy in multimodal emotion recognition. The source code is publicly available at https://figshare.com/s/8833d837871c78542b29.

show abstract

Emotion recognition in EEG signals using deep learning methods: A review

Cited by 40 publications

References 244 publications

ERTNet: an interpretable transformer-based framework for EEG emotion recognition

ERTNet: an interpretable transformer-based framework for EEG emotion recognition

EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Contact Info

Product

Resources

About