Multimodal emotional state recognition using sequence-dependent deep hierarchical features

Abstract

Emotional state recognition has become an important topic for human-robot interaction in the past years. By determining emotion expressions, robots can identify important variables of human behavior and use these to communicate in a more human-like fashion and thereby extend the interaction possibilities. Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots. Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion. Our model uses a hierarchical feature representation to deal with spontaneous emotions, and learns how to integrate multiple modalities for non-verbal emotion recognition, making it suitable to be used in an HRI scenario. Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-the-art approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.

Deep Learning for Human Affect Recognition: Insights and New Developments

“…When using 3D CNN for spatio-temporal modeling of image sequences as discussed in Section 3.1.1, the line between spatial and temporal representation learning can be blurred. While this approach is typically limited to very short sequences, with further pooling steps necessary to derive sequence-level labels (e.g., [84], [85]), in some cases spatio-temporal features can be derived for entire (short) sequences. For example, Gupta et al [62] used a variant called slow fusion [153], which treats the time domain like a spatial domain, progressively learning low-level to highlevel temporal features.…”

Developing crossmodal expression recognition based on a deep neural model

“…To be able to deal with multimodal data, our network uses the concept of the CCCNN by Barros, Jirak, Weber, and Wermter (2015a). In the CCCNN architecture, several channels, each one of them composed of an independent sequence of convolution and pooling layers, are fully connected at the end to a crosschannel layer, which is composed of convolution and pooling layers, and trained as one single architecture.…”

Survey on Emotional Body Gesture Recognition

“…Chen et al [129] used HOG on the motion history image (MHI) for finding the direction and speed, and Image-HOG features from bag of words (BOW) to compute appearance features. Another example is the usage of a multichannel CNN for learning a deep representation from the upper part of the body [130]. Finally, Botzheim et al [131] used spiking neural networks for temporal coding.…”

“…2) for learning such representations in a supervised way. Two of the very few works that uses deep learning representations for body emotion recognition are multichannel CNN from upper body [130] and spiking neural networks for temporal coding [131]. As previously discussed in Sec.…”