EEG-Based Emotion Recognition Using Convolutional Recurrent Neural Network with Multi-Head Self-Attention

Abstract

In recent years, deep learning has been widely used in emotion recognition, but the models and algorithms in practical applications still have much room for improvement. With the development of graph convolutional neural networks, new ideas for emotional recognition based on EEG have arisen. In this paper, we propose a novel deep learning model-based emotion recognition method. First, the EEG signal is spatially filtered by using the common spatial pattern (CSP), and the filtered signal is converted into a time–frequency map by continuous wavelet transform (CWT). This is used as the input data of the network; then the feature extraction and classification are performed by the deep learning model. We called this model CNN-BiLSTM-MHSA, which consists of a convolutional neural network (CNN), bi-directional long and short-term memory network (BiLSTM), and multi-head self-attention (MHSA). This network is capable of learning the time series and spatial information of EEG emotion signals in depth, smoothing EEG signals and extracting deep features with CNN, learning emotion information of future and past time series with BiLSTM, and improving recognition accuracy with MHSA by reassigning weights to emotion features. Finally, we conducted experiments on the DEAP dataset for sentiment classification, and the experimental results showed that the method has better results than the existing classification. The accuracy of high and low valence, arousal, dominance, and liking state recognition is 98.10%, and the accuracy of four classifications of high and low valence-arousal recognition is 89.33%.

Emotion Recognition Using Narrowband Spatial Features of Electroencephalography

“…The accuracy was 97.50% for SEED dataset. The CNN-BiLSTM-MHSA model [42] was proposed for emotion recognition. Here, 'The proposed study 1' which did not use CSP-generated features has 95.12% and 94.62% accuracy for valence and arousal dimensions respectively.…”

“…Accuracy (%) Valence Arousal DWT-KNN [15] 86.75 84.05 EMD -SVM [16] 70.41 72.10 MEMD -ANN [17] 72.87 75.00 CapsNet [24] 67.00 69.00 LSTM-RNN [25] 81.10 74.38 ACRNN [27] 93.72 93.38 CDCN [28] 92.24 92.92 ECLGCNN [29] 90.45 90.60 Casc-CNN-LSTM [30] 93.64 93.26 4D-CRNN [34] 94.22 94.58 PCC [37] 89.49 92.86 CNN-BiLSTM-MHSA [42] 95.12 94.62 Proposed Method 96.15 96.47…”

“…Asghar et al [26] proposed multi-modal emotion recognition approach using AlexNet for extracting time and frequency domain features and bag of deep features (BoDF) for feature reduction. Besides, attention-based convolutional recurrent neural network (ACRNN) [27], channel-fused dense convolutional network (CDCN) [28], fusion model of long-short term memories neural networks (LSTM) and graph convolutional neural network (GCNN) named ECLGCNN [29], cascaded and parallel hybrid convolution recurrent neural networks [30], dynamical graph convolutional neural networks (DGCNN) [31], spatial-temporal recurrent neural network (STRNN) [32], deep convolutional neural network [33], four-dimensional convolutional recurrent neural network (4D-CRNN) [34], graph convolutional broad network (GCB-net) along with broad learning system (BLS) [35], regularized graph neural network (RGNN) [36], combination of convolutional neural network (CNN) and deep neural network (DNN) [37], spiking neural networks (SNNs) [38], optimized residual networks (ResNet) [39], combined deep neural network (DNN) model [40], [41], CNN-BiLSTM-MHSA model [42] consisting of a convolutional neural network (CNN), bidirectional long and short-term memory network (BiLSTM), and multi-head self-attention (MHSA) were also proposed by the researchers for emotion recognition from EEG signals. Identifying EEG features that are effective for proper emotion recognition is very important.…”

Self-Supervised EEG Emotion Recognition Models Based on CNN

“…In the literature [32], the author extracted features from original EEG data and used a linear dynamic system approach to smooth these features. An average test accuracy of 87.53% was obtained by using all of the features together with a support vector machine in the SEED dataset; in the literature [15], the authors use a Discriminative Graph regularized Extreme learning machine with DE features achieves the accuracy of 91.07% for emotion classification on the SEED dataset; in the literature [33], the authors used a key channel and band detection method based on Deep Belief Network using differential entropy signals, and the final model had 86.08% classification accuracy in the SEED dataset; in the literature [34], the authors use support vector machines to classify the DEAP dataset with a final arousal classification accuracy of 64.90% and valence classification accuracy of 65.00%; in the literature [35], the authors propose a Temporal Convolutional Network and Broad Learning System, the DEAP dataset was used for the experiments, and the model achieved an average classification accuracy of 99.5755% and 99.5785% for valence and arousal, respectively. The results show that the effect of self-supervised algorithm is due to the effect of classical supervised algorithm.…”

Enhancement of Hybrid Deep Neural Network Using Activation Function for EEG based Emotion Recognition

“…In order to improve the input of Bi-LSTM network, CNN is added in the rst two layers of the model. CNN will process the signal by smoothening and reduce the sequence length so that the processed data is fed into the Bi-LSTM network instead of original data [34]. LSTM model gives good accuracy results for longer sequence of time series data.…”