Cross-Subject Multimodal Emotion Recognition Based on Hybrid Fusion

Abstract

Multimodal emotion recognition has gained traction in affective computing research community to overcome the limitations posed by the processing a single form of data and to increase recognition robustness. In this study, a novel emotion recognition system is introduced, which is based on multiple modalities including facial expressions, galvanic skin response (GSR) and electroencephalogram (EEG). This method follows a hybrid fusion strategy and yields a maximum one-subject-out accuracy of 81.2% and a mean accuracy of 74.2% on our bespoke multimodal emotion dataset (LUMED-2) for 3 emotion classes: sad, neutral and happy. Similarly, our approach yields a maximum one-subject-out accuracy of 91.5% and a mean accuracy of 53.8% on the Database for Emotion Analysis using Physiological Signals (DEAP) for varying numbers of emotion classes, 4 in average, including angry, disgust, afraid, happy, neutral, sad and surprised. The presented model is particularly useful in determining the correct emotional state in the case of natural deceptive facial expressions. In terms of emotion recognition accuracy, this study is superior to, or on par with, the reference subject-independent multimodal emotion recognition studies introduced in the literature.

Multimodal Emotion Recognition Using a Hierarchical Fusion Convolutional Neural Network

“…Taking into account that the brain regions related to the frontal lobe have high recognition accuracy [28], the 6-channel EEG signals of the forehead and the PPS signals of other remaining channels are used as experimental data in the experiment. The data is downsampled to 128 Hz, and five bands including the delta (4-8 Hz), theta (8-13 Hz), alpha (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30), beta (30)(31)(32)(33)(34)(35)(36)(37)(38)(39)(40)(41)(42)(43), and gamma bands (4-43 Hz) are filtered out. Due to the error in the first 3 s of the video in the experiment, the first 3 s of the video are removed, and the middle 30 s of the remaining duration of the video are used as experimental data.…”

“…Multimodal emotion recognition aims to combine the predictive capabilities of individual behavioral trails and biometric features for accurate classification [15]. The challenges of multimodal emotion recognition are as follows:…”

A Multimodal Deep Learning-Based Fault Detection Model for a Plastic Injection Molding Process

“…Since the early 2010s, deep neural networks, such as deep Boltzmann machine and autoencoder, have been extensively applied in order for multimodal data to effectively learn a fusion of features [42], [73]. Particularly, in domains where the data normally exist in multiple modalities, such as an activity, context recognition [74], pose estimation [75], emotion recognition [76], [77], and medical diagnosis [43], [78], deep neural networks have proven superior to other methods. Various kinds of deep neural networks are employed because they can handle multiple modalities effortlessly.…”

Analyzing online discussion data for understanding the student's critical thinking

“…Early data fusion, which integrates multiple sources of data into one feature vector, cannot fully exploit the complementary relationships among the data. Decision fusion based on the modality independence assumption makes the final decision based on multiple classifiers, but this assumption is often not true in practice (Cimtay et al , 2020). Feature fusion, which aims to combine each modality feature into one feature set, has been widely used in various classification tasks.…”