Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset

Chen, Jingxia; Jiang, Dongmei; Zhang, Yanning; Zhang, Pengwei

doi:10.1016/j.comcom.2020.02.051

Cited by 59 publications

(38 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The final accuracy reported for their multi-column model for the classification of valence and arousal is reported to be 90% and 90%, respectively. Chen et al [38] used parallel hybrid convolutional recurrent neural networks to classify the binary emotions of EEG signals. The DEAP database was also used by these researchers.…”

Section: Introductionmentioning

confidence: 99%

Recognizing Emotions Evoked by Music Using CNN-LSTM Networks on EEG Signals

et al. 2020

View full text Add to dashboard Cite

Emotion is considered to be critical for the actual interpretation of actions and relationships. Recognizing emotions from EEG signals is also becoming an important computer-aided method for diagnosing emotional disorders in neurology and psychiatry. Another advantage of this approach is recognizing emotions without clinical and medical examination, which plays a major role in completing the Brain-Computer Interface (BCI) structure. Emotions recognition ability, without traditional utilization strategies such as self-assessment tests, is of paramount importance. EEG signals are considered the most reliable technique for emotions recognition because of the non-invasive nature. Manual analysis of EEG signals is impossible for emotions recognition, so an automatic method of EEG signals should be provided for emotions recognition. One problem with automatic emotions recognition is the extraction and selection of discriminative features that generally lead to high computational complexity. This paper was design to prepare a new approach to automatic two-stage classification (negative and positive) and three-stage classification (negative, positive, and neutral) of emotions from EEG signals. In the proposed method, directly apply the raw EEG signal to the convolutional neural network and long short-term memory network (CNN-LSTM), without involving feature extraction/selection. In prior literature, this is a challenging method. The suggested deep neural network architecture includes 10-convolutional layers with 3-LSTM layers followed by 2-fully connected layers. The LSTM network in a fusion of the CNN network has been used to increase stability and reduce oscillation. In the present research, we also recorded the EEG signals of 14 subjects with music stimulation for the process. The simulation results of the proposed algorithm for two-stage classification (negative and positive) and three-stage classification (negative, neutral and positive) of emotion for 12 active channels showed 97.42% and 96.78% accuracy and Kappa coefficient of 0.94 and 0.93 respectively. We also compared our proposed LSTM-CNN network (end-toend) with other hand-crafted methods based on MLP and DBM classifiers and achieved promising results in comparison with similar approaches. According to the high accuracy of the proposed method, it can be used to develop the human-computer interface system.

show abstract

Section: Introductionmentioning

confidence: 99%

Recognizing Emotions Evoked by Music Using CNN-LSTM Networks on EEG Signals

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Similarly, Dai and colleagues [ 41 ] used a time-frequency domain representation (Spectrogram image) of the EEG obtained via the STFT. In Jingxia and colleagues [ 42 ], frequency-domain features were also used. They extracted 64 Power Spectral Density (PSD) features by using Hamming window with a width of 0.5 s in 1–47 Hz frequency.…”

Section: Resultsmentioning

confidence: 99%

Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review

et al. 2021

View full text Add to dashboard Cite

Background: Brain-Computer Interface (BCI) is becoming more reliable, thanks to the advantages of Artificial Intelligence (AI). Recently, hybrid Deep Learning (hDL), which combines different DL algorithms, has gained momentum over the past five years. In this work, we proposed a review on hDL-based BCI starting from the seminal studies in 2015. Objectives: We have reviewed 47 papers that apply hDL to the BCI system published between 2015 and 2020 extracting trends and highlighting relevant aspects to the topic. Methods: We have queried four scientific search engines (Google Scholar, PubMed, IEEE Xplore and Elsevier Science Direct) and different data items were extracted from each paper such as the database used, kind of application, online/offline training, tasks used for the BCI, pre-processing methodology adopted, type of normalization used, which kind of features were extracted, type of DL architecture used, number of layers implemented and which optimization approach were used as well. All these items were then investigated one by one to uncover trends. Results: Our investigation reveals that Electroencephalography (EEG) has been the most used technique. Interestingly, despite the lower Signal-to-Noise Ratio (SNR) of the EEG data that makes pre-processing of that data mandatory, we have found that the pre-processing has only been used in 21.28% of the cases by showing that hDL seems to be able to overcome this intrinsic drawback of the EEG data. Temporal-features seem to be the most effective with 93.94% accuracy, while spatial-temporal features are the most used with 33.33% of the cases investigated. The most used architecture has been Convolutional Neural Network-Recurrent Neural Network CNN-RNN with 47% of the cases. Moreover, half of the studies have used a low number of layers to achieve a good compromise between the complexity of the network and computational efficiency. Significance: To give useful information to the scientific community, we make our summary table of hDL-based BCI papers available and invite the community to published work to contribute to it directly. We have indicated a list of open challenges, emphasizing the need to use neuroimaging techniques other than EEG, such as functional Near-Infrared Spectroscopy (fNIRS), deeper investigate the advantages and disadvantages of using pre-processing and the relationship with the accuracy obtained. To implement new combinations of architectures, such as RNN-based and Deep Belief Network DBN-based, it is necessary to better explore the frequency and temporal-frequency features of the data at hand.

show abstract

“…Tao et al proposed an attention-based convolutional recurrent neural network (ACRNN) to extract discriminative features from EEG signals [53]. Chen et al transformed 1D chain-like EEG vector into a 2D meshlike matrix sequence [54]. The 2D matrix sequence was divided into segments containing equal time points by using sliding window, and sent to both cascaded and parallel hybrid convolutional recurrent neural network for training.…”

Section: B Deep Learning Approaches For Emotion Recognitionmentioning

confidence: 99%

Parallel Sequence-Channel Projection Convolutional Neural Network for EEG-Based Emotion Recognition

et al. 2020

View full text Add to dashboard Cite

One of the challenges in emotion recognition is finding an effective way to represent spatialtemporal features from EEG. To fully utilize the features on multiple dimensions of EEG signals, we propose a parallel sequence-channel projection convolutional neural network, including temporal stream sub-network, spatial stream sub-network, and fusion classification block. Temporal stream extracts temporal continuity via sequence-projection layer while spatial stream captures spatial correlation via channelprojection layer. Both sequence-projection and channel-projection adopt length-synchronized convolutional kernel to decode whole time and space information. The size of length-synchronized convolutional kernel is equal to the length of transmitted EEG sequence. The fusion classification block combines the extracted temporal and spatial features into a joint spatial-temporal feature vector for emotion prediction. In addition, we present a baseline noise filtering module to amplify input signals and a random channels exchange strategy to enrich the baseline-removed emotional signals. Experimental evaluation on DEAP dataset reveals that the proposed method achieves state-of-the-art classification performance for the binary classification task. The recognition accuracies reach to 96.16% and 95.89% for valence and arousal. The proposed method can improve 3% to 6% than other latest advanced works. INDEX TERMS Emotion recognition, multi-channel EEG, data augmentation, length-synchronized convolutional kernel, spatial-temporal feature.

show abstract

Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset

Cited by 59 publications

References 8 publications

Recognizing Emotions Evoked by Music Using CNN-LSTM Networks on EEG Signals

Recognizing Emotions Evoked by Music Using CNN-LSTM Networks on EEG Signals

Hybrid Deep Learning (hDL)-Based Brain-Computer Interface (BCI) Systems: A Systematic Review

Parallel Sequence-Channel Projection Convolutional Neural Network for EEG-Based Emotion Recognition

Contact Info

Product

Resources

About