Mobile Emotion Recognition via Multiple Physiological Signals using Convolution-augmented Transformer

Abstract

Recognising and monitoring emotional states play a crucial role in mental health and well-being management. Importantly, with the widespread adoption of smart mobile and wearable devices, it has become easier to collect long-term and granular emotion-related physiological data passively, continuously, and remotely. This creates new opportunities to help individuals manage their emotions and well-being in a less intrusive manner using off-the-shelf low-cost devices. Pervasive emotion recognition based on physiological signals is, however, still challenging due to the difficulty to efficiently extract high-order correlations between physiological signals and users' emotional states. In this paper, we propose a novel end-to-end emotion recognition system based on a convolution-augmented transformer architecture. Specifically, it can recognise users' emotions on the dimensions of arousal and valence by learning both the global and local fine-grained associations and dependencies within and across multimodal physiological data (including blood volume pulse, electrodermal activity, heart rate, and skin temperature). We extensively evaluated the performance of our model using the K-EmoCon dataset, which is acquired in naturalistic conversations using off-the-shelf devices and contains spontaneous emotion data. Our results demonstrate that our approach outperforms the baselines and achieves state-of-the-art or competitive performance. We also demonstrate the effectiveness and generalizability of our system on another affective dataset which used affect inducement and commercial physiological sensors.

LSTM-Modeling of Emotion Recognition Using Peripheral Physiological Signals in Naturalistic Conversations

“…Table VI shows the baseline classification results for arousal, valence, and four quadrants in comparison with the proposed method, as well as state-of-the-art methods implemented on the K-EmoCon dataset [17], [40]. Accuracy (Acc.)…”

Approaches, Applications, and Challenges in Physiological Emotion Recognition—A Tutorial Overview

“…They have been first implemented for natural language processing (NLP) and employ attention mechanisms to analyze sequences of words and are appropriate for use in other applications, such as time-series forecasting, medical, physiological signal analysis, and human activity recognition [155]. Recent studies also use these architectures for recognizing emotions from physiological signals (see [156] and [155]). Yang et al [156] combined CNN architectures with conformer blocks and tested them on PPG, EDA, and ST data from the K-Emocon dataset.…”

“…Recent studies also use these architectures for recognizing emotions from physiological signals (see [156] and [155]). Yang et al [156] combined CNN architectures with conformer blocks and tested them on PPG, EDA, and ST data from the K-Emocon dataset. They achieved 77.37% and 79.42% accuracies for detecting valence and arousal levels.…”

DREAM: Deep Learning-Based Recognition of Emotions From Multiple Affective Modalities Using Consumer-Grade Body Sensors and Video Cameras

“…Chowdhary et al in 2021 used ConvNet on pre-trained models for identifying emotions from facial expressions in CK+ dataset [21]. Ahmed et al and Zhang et al have worked on emotion recognition from speech signals, by analysing the linguistic parameters, and the melogram pattern of the speech signals[32,33].K-EmoCon dataset was published in 2020, only few researchers have worked on K-EmoCon till now, none of whom used all the modalities of the dataset for emotion recognition. Gupta et al[22], Zitouni et al[23], Alskafi.…”

“…Quan et al used only audio and visual signals of 16 participants for emotion recognition[18]. Yang et al has used CNN on K-EmoCon for identifying emotion recognition using physiological signals only, not the face expressions or linguistic patterns at all[33]. Alhussein used CNN on only linguistic and speech signals of K-EmoCon to identify the emotions in arousal and valance dimensional emotions, having state of the art f1score of 82%[34].…”