Interpretable Emotion Recognition Using EEG Signals

Qing, Chunmei; Qiao, Rui; Xu, Xiangmin; Cheng, Yongqiang

doi:10.1109/access.2019.2928691

Cited by 115 publications

(49 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Several unimodal and multimodal emotion recognition studies have been reported in the literature [12][13][14][15][16]. Majority of studies use non-physiological data, such as audio, video and text [17].…”

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

2020

View full text Add to dashboard Cite

Multimodal emotion recognition has gained traction in affective computing research community to overcome the limitations posed by the processing a single form of data and to increase recognition robustness. In this study, a novel emotion recognition system is introduced, which is based on multiple modalities including facial expressions, galvanic skin response (GSR) and electroencephalogram (EEG). This method follows a hybrid fusion strategy and yields a maximum one-subject-out accuracy of 81.2% and a mean accuracy of 74.2% on our bespoke multimodal emotion dataset (LUMED-2) for 3 emotion classes: sad, neutral and happy. Similarly, our approach yields a maximum one-subject-out accuracy of 91.5% and a mean accuracy of 53.8% on the Database for Emotion Analysis using Physiological Signals (DEAP) for varying numbers of emotion classes, 4 in average, including angry, disgust, afraid, happy, neutral, sad and surprised. The presented model is particularly useful in determining the correct emotional state in the case of natural deceptive facial expressions. In terms of emotion recognition accuracy, this study is superior to, or on par with, the reference subject-independent multimodal emotion recognition studies introduced in the literature.

show abstract

Section: Background and Literature Review On Multimodal Emotion Rmentioning

confidence: 99%

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

2020

View full text Add to dashboard Cite

show abstract

“…Many physiological modalities and features have been evaluated for ER, namely Electroencephalography (EEG) [ 28 , 29 , 30 ], Electrocardiography (ECG) [ 31 , 32 , 33 ], Electrodermal Activity (EDA) [ 34 , 35 , 36 ], Respiration (RESP) [ 26 ], Blood Volume Pulse (BVP) [ 26 , 35 ] and Temperature (TEMP) [ 26 ]. Multi-modal approaches have prevailed; however, there is still no clear evidence of which feature combinations and physiological signals are the most relevant.…”

Section: State Of the Artmentioning

confidence: 99%

Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

Bota

Wang²,

Fred

et al. 2020

Sensors

View full text Add to dashboard Cite

Emotion recognition based on physiological data classification has been a topic of increasingly growing interest for more than a decade. However, there is a lack of systematic analysis in literature regarding the selection of classifiers to use, sensor modalities, features and range of expected accuracy, just to name a few limitations. In this work, we evaluate emotion in terms of low/high arousal and valence classification through Supervised Learning (SL), Decision Fusion (DF) and Feature Fusion (FF) techniques using multimodal physiological data, namely, Electrocardiography (ECG), Electrodermal Activity (EDA), Respiration (RESP), or Blood Volume Pulse (BVP). The main contribution of our work is a systematic study across five public datasets commonly used in the Emotion Recognition (ER) state-of-the-art, namely: (1) Classification performance analysis of ER benchmarking datasets in the arousal/valence space; (2) Summarising the ranges of the classification accuracy reported across the existing literature; (3) Characterising the results for diverse classifiers, sensor modalities and feature set combinations for ER using accuracy and F1-score; (4) Exploration of an extended feature set for each modality; (5) Systematic analysis of multimodal classification in DF and FF approaches. The experimental results showed that FF is the most competitive technique in terms of classification accuracy and computational complexity. We obtain superior or comparable results to those reported in the state-of-the-art for the selected datasets.

show abstract

“…The real-world impact of BCIs is promising because they can identify intention-reflected brain activities. In the past decade, human-centered BCIs, such as those in mental fatigue detection tasks (Binias et al, 2020 ; Ko et al, 2020b ), emotion recognition (Qing et al, 2019 ), and controlling exoskeletons (Lee et al, 2017 ) have shed light on the success of improving human ability. An active BCI (Fahimi et al, 2020 ) recognizes complex patterns from EEG spontaneously caused by a user's intention independent of external stimuli, and a reactive BCI (Won et al, 2019 ) identifies brain activities in reaction to external events.…”

Section: Introductionmentioning

confidence: 99%

A Survey on Deep Learning-Based Short/Zero-Calibration Approaches for EEG-Based Brain–Computer Interfaces

Jeon

Jeong

et al. 2021

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Brain–computer interfaces (BCIs) utilizing machine learning techniques are an emerging technology that enables a communication pathway between a user and an external system, such as a computer. Owing to its practicality, electroencephalography (EEG) is one of the most widely used measurements for BCI. However, EEG has complex patterns and EEG-based BCIs mostly involve a cost/time-consuming calibration phase; thus, acquiring sufficient EEG data is rarely possible. Recently, deep learning (DL) has had a theoretical/practical impact on BCI research because of its use in learning representations of complex patterns inherent in EEG. Moreover, algorithmic advances in DL facilitate short/zero-calibration in BCI, thereby suppressing the data acquisition phase. Those advancements include data augmentation (DA), increasing the number of training samples without acquiring additional data, and transfer learning (TL), taking advantage of representative knowledge obtained from one dataset to address the so-called data insufficiency problem in other datasets. In this study, we review DL-based short/zero-calibration methods for BCI. Further, we elaborate methodological/algorithmic trends, highlight intriguing approaches in the literature, and discuss directions for further research. In particular, we search for generative model-based and geometric manipulation-based DA methods. Additionally, we categorize TL techniques in DL-based BCIs into explicit and implicit methods. Our systematization reveals advances in the DA and TL methods. Among the studies reviewed herein, ~45% of DA studies used generative model-based techniques, whereas ~45% of TL studies used explicit knowledge transferring strategy. Moreover, based on our literature review, we recommend an appropriate DA strategy for DL-based BCIs and discuss trends of TLs used in DL-based BCIs.

show abstract

Interpretable Emotion Recognition Using EEG Signals

Cited by 115 publications

References 29 publications

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Emotion Assessment Using Feature Fusion and Decision Fusion Classification Based on Physiological Data: Are We There Yet?

A Survey on Deep Learning-Based Short/Zero-Calibration Approaches for EEG-Based Brain–Computer Interfaces

Contact Info

Product

Resources

About