Emotion recognition based on multi-modal physiological signals and transfer learning

Abstract

In emotion recognition based on physiological signals, collecting enough labeled data of a single subject for training is time-consuming and expensive. The physiological signals’ individual differences and the inherent noise will significantly affect emotion recognition accuracy. To overcome the difference in subject physiological signals, we propose a joint probability domain adaptation with the bi-projection matrix algorithm (JPDA-BPM). The bi-projection matrix method fully considers the source and target domain’s different feature distributions. It can better project the source and target domains into the feature space, thereby increasing the algorithm’s performance. We propose a substructure-based joint probability domain adaptation algorithm (SSJPDA) to overcome physiological signals’ noise effect. This method can avoid the shortcomings that the domain level matching is too rough and the sample level matching is susceptible to noise. In order to verify the effectiveness of the proposed transfer learning algorithm in emotion recognition based on physiological signals, we verified it on the database for emotion analysis using physiological signals (DEAP dataset). The experimental results show that the average recognition accuracy of the proposed SSJPDA-BPM algorithm in the multimodal fusion physiological data from the DEAP dataset is 63.6 and 64.4% in valence and arousal, respectively. Compared with joint probability domain adaptation (JPDA), the performance of valence and arousal recognition accuracy increased by 17.6 and 13.4%, respectively.

Towards non-invasive PTSD diagnosis: Utilising EEG based Emotion Recognition with the DEAP Database

“…A novel method based on substructure-based joint probability domain adaptation was created by Fu et al [15] to lessen the detrimental effects of noise on physiological inputs. Pusarla et al [16] improved upon earlier approaches in terms of emotion identi cation accuracy by employing a local mean decomposition methodology for EEG signals.…”

Survey of Transfer Learning Approaches in the Machine Learning of Digital Health Sensing Data

“…The authors demonstrated that the learned mappings reflected prior knowledge on the semantic relationships between domains, making the MSADA a powerful tool for exploratory activity data analysis; the proposed multisource domain adaptation approach achieved a 2% and a 13% improvement in accuracy on both the OPPORTUNITY dataset and the DSADS dataset, respectively. In [221], the authors proposed a new method called joint probability domain adaptation with a bi-projection matrix algorithm (JPDA-BPM) to overcome the challenge of collecting enough labeled data for emotion recognition based on physiological signals, which is time-consuming and expensive. The proposed method considered the differences in feature distributions between the source and target domains, which improved the algorithm's performance.…”

A Transfer Learning-based Approach for Multimodal Emotion Recognition

“…For the MSP-IMPROV dataset, the suggested method obtained results that are considered to be state-of-the-art. Using 3D convolutional neural networks, the authors [17] proposed a transfer learning approach for multimodal emotion recognition (CNNs). For the recognition of facial expressions, they used pre-trained models and then fine-tuned them using their own dataset.…”