Combining Inter-Subject Modeling with a Subject-Based Data Transformation to Improve Affect Recognition from EEG Signals

Arevalillo‐Herráez, Miguel; Cobos, Máximo; Roger, Sandra; García-Pineda, Miguel

doi:10.3390/s19132999

Cited by 33 publications

(29 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, an interesting question to be asked is at what depth of the neural network does stratified normalization help increase the accuracy. To answer this question, we further analyzed the emotion recognition accuracy of the models, and their ability to capture and exploit the brain signature of each participant – defined as the part of information extracted from the brain signals that is specific to that participant (also called subject-related component by Arevalillo-Herráez et al (2019)), such that it can directly inform us on which participant it’s been extracted from . Intuitively, we would expect the emotion recognition accuracy to increase with each layer of the neural network, while the brain signature would fade out due to a decrease of the inter-participant variability with each data normalization.…”

Section: Resultsmentioning

confidence: 99%

“…The studies by Koelstra et al (2012) and Jatupaiboon et al (2013) already gave the first insights into the advantages of this approach after applying participant-based data normalization to reduce the inter-participant variability. Later on, the work of Arevalillo-Herráez et al (2019) exploited this result and proposed a nonlinear data transformation that seamlessly integrated individual traits into an inter-participant approach.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Subject EEG-Based Emotion Recognition through Neural Networks with Stratified Normalization

Fernández¹,

Guttenberg²,

Witkowski³

et al. 2020

Preprint

View full text Add to dashboard Cite

Due to a large number of potential applications, a good deal of effort has been recently made towards creating machine learning models that can recognize evoked emotions from one’s physiological recordings. In particular, researchers are investigating the use of EEG as a low-cost, non-invasive method. However, the poor homogeneity of the EEG activity across participants hinders the implementation of any such system by a time-consuming calibration stage. In this study, we introduce a new participant-based feature normalization method, so-called stratified normalization, for training deep neural networks in task of cross-subject emotion classification from EEG signals. The new method is able to subtract inter-participant variability while maintaining the emotion information in the data. We carried out our analysis on the SEED dataset, which contains 62-channel EEG recordings collected from 15 participants while watching film clips. Results demonstrate that networks trained with stratified normalization outperformed standard training with batch normalization significantly. In addition, the highest model performance was achieved when extracting EEG features with the multitaper method, reaching a classification accuracy of 91.6% for two emotion categories (positive and negative) and 79.6% for three (also neutral). This analysis provides us with great insight into the potential benefits that stratified normalization can have when developing any cross-subject model based on EEG.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Cross-Subject EEG-Based Emotion Recognition through Neural Networks with Stratified Normalization

Fernández¹,

Guttenberg²,

Witkowski³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…However, in the case of four classes of emotions (HVHA, HVLA, LVHA, and LVLA) the reported recognition rates decrease to a much larger extent [17]. Fourth, due to the highly subjective and dynamic nature of emotion, few pieces of research have proposed subject-dependent approaches with better recognition performance; however, subject-dependent approaches besides higher recognition rates require large training data from each subject and are not capable of being applied to unseen subjects [46]. Therefore, studies reported for subject-independent approaches [47,48] are more reliable for emotion elicitation tasks from a broader perspective as compared to the results obtained using subject-dependent approaches [49,50].…”

Section: Discussionmentioning

confidence: 99%

“…The second category with specialized pre-processing consists of steps required specific to the emotion elicitation algorithm. These steps include baseline removal and Z-score normalization, inspired by its significance in many studies such as [46,[58][59][60][61].…”

Section: Pre-processingmentioning

confidence: 99%

CNN and LSTM-Based Emotion Charting Using Physiological Signals

Dar

Akram

Khawaja

et al. 2020

Sensors

View full text Add to dashboard Cite

Novel trends in affective computing are based on reliable sources of physiological signals such as Electroencephalogram (EEG), Electrocardiogram (ECG), and Galvanic Skin Response (GSR). The use of these signals provides challenges of performance improvement within a broader set of emotion classes in a less constrained real-world environment. To overcome these challenges, we propose a computational framework of 2D Convolutional Neural Network (CNN) architecture for the arrangement of 14 channels of EEG, and a combination of Long Short-Term Memory (LSTM) and 1D-CNN architecture for ECG and GSR. Our approach is subject-independent and incorporates two publicly available datasets of DREAMER and AMIGOS with low-cost, wearable sensors to extract physiological signals suitable for real-world environments. The results outperform state-of-the-art approaches for classification into four classes, namely High Valence—High Arousal, High Valence—Low Arousal, Low Valence—High Arousal, and Low Valence—Low Arousal. Emotion elicitation average accuracy of 98.73% is achieved with ECG right-channel modality, 76.65% with EEG modality, and 63.67% with GSR modality for AMIGOS. The overall highest accuracy of 99.0% for the AMIGOS dataset and 90.8% for the DREAMER dataset is achieved with multi-modal fusion. A strong correlation between spectral- and hidden-layer feature analysis with classification performance suggests the efficacy of the proposed method for significant feature extraction and higher emotion elicitation performance to a broader context for less constrained environments.

show abstract

“…Another early research approach is to build individual models adapted to each user, known as subject-dependent (intra-subject) models. The benefits of the subject-dependent models have recently been proved in several cases such as Electroencephalography (EEG) signals in [20][21][22] or keyboard and mouse signals in [23]. This approach offers better detection accuracy, by offering models adapted to each subject.…”

Section: Introductionmentioning

confidence: 99%

An Intra-Subject Approach Based on the Application of HMM to Predict Concentration in Educational Contexts From Non-Intrusive Physiological Signals Inreal-World Situations

Serrano-Mamolar¹,

Arevalillo‐Herráez²,

Boticario³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Emotion recognition is becoming very relevant in educational scenarios, since previous research has proven the strong influence of emotions on the student's engagement and motivation. There is no standard method for stating student's affect, but physiological signals have been widely used in educational contexts. Physiological signals have been proved to offer high accuracy in detecting emotions because they reflect spontaneous affect-related information, and which is fresh and do not require an additional control or interpretation. However, most proposed works use measuring equipment that limit its applicability in real-world scenarios because of their high cost and their intrusiveness. Expensive material means an economic challenge for schools and reduce the scalability of the experiments. Intrusive equipment can be uncomfortable for the students which can lead to errors in the collected data. In this work, we analyse the feasibility of developing a low-cost non-intrusive device that integrates easy-to-capture signals that guarantee high detection accuracy. The advantage of the approach also lies in using user’s centred information sources (intra-subject) in real-world situations, which provide better detection accuracy, by offering models adapted to each subject. To this end, we present an experimental study that aims to explore the potential application of Hidden Markov Models (HMM) to predict the concentration state from 4 commonly used physiological signals, namely heart rate, breath rate, skin conductance and skin temperature. We study the multi-fusion of every possible combination of these four signals and analyse their potential use in an educational context in terms of intrusiveness, cost and accuracy. Results show that a high accuracy can be achieved with three of the signals when using HMM-based intra-subject models. However, inter-subject models, which are meant to obtain subject-independent approaches for affect detection, fail at the same task. This work concludes that the implementation of a low-cost wrist-worn device for recognising relevant emotions from each student is possible and open the way to a wide range of practical applications in the context of adaptive learning systems.

show abstract

Combining Inter-Subject Modeling with a Subject-Based Data Transformation to Improve Affect Recognition from EEG Signals

Cited by 33 publications

References 68 publications

Cross-Subject EEG-Based Emotion Recognition through Neural Networks with Stratified Normalization

Cross-Subject EEG-Based Emotion Recognition through Neural Networks with Stratified Normalization

CNN and LSTM-Based Emotion Charting Using Physiological Signals

An Intra-Subject Approach Based on the Application of HMM to Predict Concentration in Educational Contexts From Non-Intrusive Physiological Signals Inreal-World Situations

Contact Info

Product

Resources

About