Recognition of emotions using multimodal physiological signals and an ensemble deep learning model

Yin, Zhong; Zhao, Mengyuan; Wang, Yongxiong; Yang, Jingdong; Zhang, Jianhua

doi:10.1016/j.cmpb.2016.12.005

Cited by 315 publications

(121 citation statements)

References 43 publications

Supporting

Mentioning

117

Contrasting

Order By: Relevance

“…Deep learning has also been used as part of ensemble methods [169], and in the form of Echo State Networks [170] for dimensionality reduction of handcrafted EEG features. Yin et al [171] used stacked autoencoders (SAEs) to learn high-level representations from various peripheral sensors including skin temperature and blood volume pressure, improving the state-of-the-art by 5%.…”

Section: Learning Temporal Features From Physiological Datamentioning

confidence: 99%

See 1 more Smart Citation

Deep Learning for Human Affect Recognition: Insights and New Developments

Rouast

Adam

Chiong

2021

IEEE Trans. Affective Comput.

130

View full text Add to dashboard Cite

Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications.

show abstract

Section: Learning Temporal Features From Physiological Datamentioning

confidence: 99%

“…Feature-level fusion can also be based solely on physiological measurements. Yin et al [171] successfully used a fusion SAE to aggregate handcrafted features from several different sensors. Similarly, Liu et al [175] used handcrafted features derived from EEG and eye tracking as input into a SAE.…”

Section: Learning Joint Features With Physiological Datamentioning

confidence: 99%

Deep Learning for Human Affect Recognition: Insights and New Developments

Rouast

Adam

Chiong

2021

IEEE Trans. Affective Comput.

130

View full text Add to dashboard Cite

show abstract

“…An auto-encoder (AE) based on Restricted Boltzmann Machine (RBM) is used to fuse the two kinds of signals, and then SVM is applied to recognize emotion in terms of valence. Yin et al [9] also conduct emotion recognition with a single-type EP signal and a single-channel EEG signal. The difference is that the signals are normalized at first and an AE based on fully connected neural networks (FCNNs) is used for feature fusion.…”

Section: Introductionmentioning

confidence: 99%

“…Zhuang et al[6] Liu et al[8] Yin et al[9] Cheng et al[27] ; mRMR, minimum redundancy-maximum-relevance; EOG, electrooculogram; SVM, support vector machine; GSR, galvanic skin response; FCNN, fully connected neural network.…”

mentioning

confidence: 99%

Multimodal Emotion Recognition Method Based on Convolutional Auto-Encoder

Zhou¹,

Wei²,

Cheng³

et al. 2018

IJCIS

View full text Add to dashboard Cite

Emotion recognition is of great significance to computational intelligence systems. In order to improve the accuracy of emotion recognition, electroencephalogram (EEG) signals and external physiological (EP) signals are adopted due to their perfect performance in reflecting the slight variations of emotions, wherein EEG signals consist of multiple channels signals and EP signals consist of multiple types of signals. In this paper, a multimodal emotion recognition method based on convolutional auto-encoder (CAE) is proposed. Firstly, a CAE is designed to obtain the fusion features of multichannel EEG signals and multitype EP signals. Secondly, a fully connected neural network classifier is constructed to achieve emotion recognition. Finally, experiment results show that the proposed method can improve the accuracy of emotion recognition obviously compared with other similar methods.

show abstract

“…Yin, et al [21] introduced a Multiple fusion layer based Ensemble Classifier of Stacked Auto Encoder (MESAE) for recognizing the emotions of the facial image. Here, the over fitting was avoided by estimating the artificial feature vector.…”

Section: Related Workmentioning

confidence: 99%

CLBPGNN Convoluted Local Binary Pattern based Grouping Neural Network for Face Emotion Recognition

G.¹,

Krithika²

2017

JESTR

View full text Add to dashboard Cite

Due to the variations of facial expressions, the process of emotion recognition is a highly demanding and challenging task in many image processing applications. So, the traditional works introduced different techniques for facial image recognitions, which includes the drawbacks of reduced accuracy and recognition rate. Thus, this work aims to develop a new emotion recognition system by implementing different image processing techniques. Initially, the image obtained from the video sequence is preprocessed by employing the Neighborhood Pixel Intensity based Filtering (NPIF) technique. Then, the Viola Jones algorithm is implemented to extract the face by suppressing the background and foreground regions, which increases the accuracy of recognition. After that, the Convoluted Local Binary Pattern (CLBP) technique is employed to extract the patterns of the background suppressed image. Based on the extracted key points, the emotion of the given image is recognized by using the Grouping based Neural Network (GNN) classification technique.

show abstract

Recognition of emotions using multimodal physiological signals and an ensemble deep learning model

Cited by 315 publications

References 43 publications

Deep Learning for Human Affect Recognition: Insights and New Developments

Deep Learning for Human Affect Recognition: Insights and New Developments

Multimodal Emotion Recognition Method Based on Convolutional Auto-Encoder

CLBPGNN Convoluted Local Binary Pattern based Grouping Neural Network for Face Emotion Recognition

Contact Info

Product

Resources

About