Expression-EEG Based Collaborative Multimodal Emotion Recognition Using Deep AutoEncoder

Abstract

Emotion recognition has shown many valuable roles in people's lives under the background of artificial intelligence technology. However, most existing emotion recognition methods have poor recognition performance, which prevents their promotion in practical applications. To alleviate this problem, we proposed an expression-EEG interaction multi-modal emotion recognition method using a deep automatic encoder. Firstly, decision tree is applied as objective feature selection method. Then, based on the facial expression features recognized by sparse representation, the solution vector coefficients are analyzed to determine the facial expression category of the test samples. After that, the bimodal deep automatic encoder is adopted to fuse the EEG signals and facial expression signals. The third layer of BDAE extracts features for training of supervised learning. Finally, LIBSVM classifier is used to complete classification task. We carried out experiments on a constructed video library to verify the proposed emotion recognition method. The results show that the proposed method can effectively extract and integrate high-level emotion-related features in EEG and facial expression signals. The recognition rate of discrete emotion state type and the average emotion recognition rate have been improved relatively, in which the average emotion recognition rate is 85.71%. Overall, the emotion recognition ability has been greatly improved.

Multimodal Emotion Recognition using Deep Learning

“…At present, This technique was supplemented to increase the precision of emotion detection further. Multimodal fusion model can achieve emotion detection results by integrating physiological signals in various ways [58]. With the recent developments in Deep Learning (DL) architectures, deep learning has been applied [59] in multimodal emotion recognition.…”

“…However, when only two physiological signals (EEG,BVP) are considered, the classification accuracy was 71.61%. In [58], [62], they both used CNN, and two physiological signals are considered, but in [62], more than one classification algorithms are used, which had a great impact on the results with an accuracy rate of more than 97% on SEED database. From here, we noticed that with more than one algorithm, the results are better in some cases.…”

Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network

“…Taking into account that the brain regions related to the frontal lobe have high recognition accuracy [28], the 6-channel EEG signals of the forehead and the PPS signals of other remaining channels are used as experimental data in the experiment. The data is downsampled to 128 Hz, and five bands including the delta (4-8 Hz), theta (8-13 Hz), alpha (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30), beta (30)(31)(32)(33)(34)(35)(36)(37)(38)(39)(40)(41)(42)(43), and gamma bands (4-43 Hz) are filtered out. Due to the error in the first 3 s of the video in the experiment, the first 3 s of the video are removed, and the middle 30 s of the remaining duration of the video are used as experimental data.…”

“…Deep learning results show that automatic feature extraction performs better than manual feature extraction; and various deep learning technologies [10], including autoencoders (AEs), convolutional neural networks (CNNs) [11], and recurrent neural networks (RNNs) [12], are widely used in different domains. Among these technologies, CNNs have the ability to find robust spatial features from images, RNNs are suitable for extracting the temporal features of video and speech for classification, and AEs are more suitable for learning unsupervised features [13]. In a CNN, each CNN layer contains some relevant features that represent important information at their respective level of abstraction of the input data.…”

TSN: Performance Creative Choreography Based on Twin Sensor Network

“…Using samples of 55 healthy subjects, Pinto et al used single-peak and multipeak methods to analyse which signals or combinations of signals can better describe emotional responses. Zhang [29] proposed a multimodal emotion recognition method using deep autoencoders for facial expressions and EEG interactions. The decision tree is used as the target feature selection method.…”