Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data

Abstract

To further extend the applicability of wearable sensors in various domains such as mobile health systems and the automotive industry, new methods for accurately extracting subtle physiological information from these wearable sensors are required. However, the extraction of valuable information from physiological signals is still challenging—smartphones can count steps and compute heart rate, but they cannot recognize emotions and related affective states. This study analyzes the possibility of using end-to-end multimodal deep learning (DL) methods for affect recognition. Ten end-to-end DL architectures are compared on four different datasets with diverse raw physiological signals used for affect recognition, including emotional and stress states. The DL architectures specialized for time-series classification were enhanced to simultaneously facilitate learning from multiple sensors, each having their own sampling frequency. To enable fair comparison among the different DL architectures, Bayesian optimization was used for hyperparameter tuning. The experimental results showed that the performance of the models depends on the intensity of the physiological response induced by the affective stimuli, i.e., the DL models recognize stress induced by the Trier Social Stress Test more successfully than they recognize emotional changes induced by watching affective content, e.g., funny videos. Additionally, the results showed that the CNN-based architectures might be more suitable than LSTM-based architectures for affect recognition from physiological sensors.

The Cold Start Problem and Per-Group Personalization in Real-Life Emotion Recognition With Wearables

“…For FCN and ResNet, all parts of a window were treated as separate channels (3 window parts × 4 signals = 12 channels in total). The deep learning architectures were programmed in PyTorch [33] according to an article by Dzieżyc et al [34]. For classical machine learning algorithms, we used implementations from scikit-learn [35].…”

Subject independent emotion recognition using EEG and physiological signals – a comparative study

“…The body of work in literature has explored the feature extraction ability of deep learning networks for end-to-end ER architectures and its performance was determined by the strength of the input signals [11]. Deep learning architectures like ensemble convolution neural network (ECNN) [5], DBN [6], inception ResNet v2 [12], spiking neural networks (SNN) [13], autoencoder [14], hierarchy modular neural network (HMNN) [15], MDBN [16], transfer learning [17], transformer-based architecture using CNN [17] and high resolution network (HRNet) [18] were explored for ER.…”

Tutorial on Deep Learning for Human Activity Recognition

“…At the latest since deep learning has found its way into activity recognition, the standards of former algorithms are often no longer completely applicable. Feature engineering, as known from classical machine learning approaches, is no longer necessary since [14] showed that raw sensor data can be processed by using deep neural networks, even though, recent publications like [3] show that it can be beneficial when it comes to specific domains and circumstances or when using specific architectures of neural networks [6]. In classical machine learning, however, statistics of the first order are mostly used (mean, variance, median values, etc.).…”