1D Convolutional Autoencoder-Based PPG and GSR Signals for Real-Time Emotion Classification

Abstract

To apply emotion recognition and classification technology to the field of human-robot interaction, it is necessary to implement fast data processing and model weight reduction. This paper proposes a new photoplethysmogram (PPG) and galvanic skin response (GSR) signals-based labeling method using Asian multimodal data, a real-time emotion classification method, a 1d convolutional neural network autoencoder model, and a lightweight model obtained using knowledge distillation. In addition, the model performance was verified using the public DEAP dataset and the Asian multi-modal dataset 'MERTI-Apps'. For emotion classification, bio-signal data were window-sliced in 1-pulse units, and the label was reset to reflect the characteristics of the PPG and GSR signals. Simple data pre-processing, such as the prevention of loss and waveform duplication, was performed without using handcrafted features. The experiment showed that the accuracy of the proposed model using MERTI-Apps was 79.18% and 74.84% in the case of arousal and valence, respectively, for 3-class criteria, and the accuracy of the proposed model using DEAP was 81.33% and 80.25% in the case of arousal and valence, respectively, for 2-class criteria. The accuracy of the lightweight model was 77.87% and 73.49% in the case of arousal and valence, respectively, for 3-class criteria and its calculation time was reduced by more than 80% compared to the proposed 1d convolutional autoencoder model. We also confirmed that the proposed model improved computational time and accuracy compared to previous studies using MERTI-Apps and the lightweight model used in limited hardware environments enabled fast computation and real-time emotion classification.

Emotion Classification from Multi-Band Electroencephalogram Data Using Dynamic Simplifying Graph Convolutional Network and Channel Style Recalibration Module

“…For example, Li et al proposed a semi-supervised deep facial expression recognition method based on an adaptive confidence margin [ 2 ]. Kang et al proposed a one-dimensional convolutional autoencoder to classify emotions by PPG and GSR [ 3 ]. In contrast, it is unfriendly to use non-physiological signals to identify emotions for special people or people with facial, limb, and voice damage caused by accidents.…”

Enabling Remote Responder Bio-Signal Monitoring in a Cooperative Human–Robot Architecture for Search and Rescue

“…Moreover, near-fall detection for the elderly and people with Parkinson’s disease using EEG and EMG [ 27 ] and machine learning based on stroke disease prediction using ECG and photoplethysmography (PPG) [ 28 ] are examples of applications in medical settings. Another specific application is real-time emotion classification, where [ 29 ] proposed a convolutional autoencoder based on PPG and GSR signals from experiments where subjects watched a short video and marked self-assessment labeling of two or three classes (positive, negative or neutral emotions). Similarly, Ref.…”