Context-Aware Emotion Recognition Networks

Abstract

Traditional techniques for emotion recognition have focused on the facial expression analysis only, thus providing limited ability to encode context that comprehensively represents the emotional responses. We present deep networks for context-aware emotion recognition, called CAER-Net, that exploit not only human facial expression but also context information in a joint and boosting manner. The key idea is to hide human faces in a visual scene and seek other contexts based on an attention mechanism. Our networks consist of two sub-networks, including two-stream encoding networks to separately extract the features of face and context regions, and adaptive fusion networks to fuse such features in an adaptive fashion. We also introduce a novel benchmark for context-aware emotion recognition, called CAER, that is more appropriate than existing benchmarks both qualitatively and quantitatively. On several benchmarks, CAER-Net proves the effect of context for emotion recognition. Our dataset is available at

Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

“…Moreover, the rapid development and mass production of consumer grade devices, such as smartband and smartwatch (Poongodi et al, 2020), will facilitate the integration of these signals in most of HRI systems. For example, in Lazzeri et al (2014) a multimodal acquisition platform, including a humanoid robot capable of expressing emotions, was tested in a social robot-based therapy scenario for children with autism. The system included audio and video sources, together with electrocardiogram (ECG), GSR, respiration, and accelerometer, that are integrated in a sensorized t-shirt, but the platform was designed to be flexible and reconfigurable in order to connect with various hardware devices.…”

Emotion Recognition on FER-2013 Face Images Using Fine-Tuned VGG-16

“…These researches were conducted with broad and separate focus, such as establishing dataset [11,12,25], mapping features or feature selection process(es) [26,27], designing architecture [27,28], and adopting approaches from other fields of study [25,27]. More recent studies adopted CNN for face emotion recognition, such as the approaches proposed in [29][30][31] One, if not the most, attractive feature of emotion recognition is the human face itself. The main reason behind this is because mostly human face expresses emotions one feels [32].…”

Graph Reasoning-Based Emotion Recognition Network

“…In recent years, Convolutional Neural Network (CNN)based methods [14]- [16] have achieved more accurate and robust emotion recognition than previous methods with changes in surrounding information. Yu et al [15] created a model to recognize the emotions of static images, which contains three face detectors and a multiple deep CNNs module.…”

“…In visual emotion recognition, other visual cues such as body gestures, actions, and environmental contexts can show additional useful information. Thus, Lee et al [14] integrated the facial expressions and surrounding information of people with adaptive fusion networks to demonstrate that the performance of emotion recognition networks can be remarkably boosted by integrating facial and context information. In fact, the emotion recognition system can analyze features at the local pixel level, which are extracted by a specific convolution receptive field.…”

“…In specific, our proposed GCN-based emotion recognition framework has the following major contributions: 1) Unlike the traditional CNN-based methods that simply consider the context of images or video frame input, our proposed network recognizes emotions from context-aware emotion datasets with reasonable and filter mechanism, which can reason the relationships of different regions and remove redundant information among node vectors. 2) This work examines distinct approaches to verify that the proposed framework extracts spatiotemporal features more efficiently than previous methods over the Context-aware Emotion Recognition (CAER) [14] and Acted Facial Expressions in the Wild (AFEW) [19] datasets, which includes the complex context information.…”