Developing crossmodal expression recognition based on a deep neural model

Abstract

A robot capable of understanding emotion expressions can increase its own capability of solving problems by using emotion expressions as part of its own decision-making, in a similar way to humans. Evidence shows that the perception of human interaction starts with an innate perception mechanism, where the interaction between different entities is perceived and categorized into two very clear directions: positive or negative. While the person is developing during childhood, the perception evolves and is shaped based on the observation of human interaction, creating the capability to learn different categories of expressions. In the context of human–robot interaction, we propose a model that simulates the innate perception of audio–visual emotion expressions with deep neural networks, that learns new expressions by categorizing them into emotional clusters with a self-organizing layer. The proposed model is evaluated with three different corpora: The Surrey Audio–Visual Expressed Emotion (SAVEE) database, the visual Bi-modal Face and Body benchmark (FABO) database, and the multimodal corpus of the Emotion Recognition in the Wild (EmotiW) challenge. We use these corpora to evaluate the performance of the model to recognize emotional expressions, and compare it to state-of-the-art research.

Affective Touch in Human–Robot Interaction: Conveying Emotion to the Nao Robot

“…[5]), including with reference to decoding tactile (gestural) inputs [12]. Furthermore, Hertenstein et al [25] found a lower mean percentage correct classification/decoding for the Rejection (64%) and Attachment (59%) emotions identified above, than we did in our study.…”

“…The reason for this was that the instructions vocalized by the experimenters did not request time-limited responding from participants regarding the conveyed emotions. Time-limitation was considered 5 We winsorized 3 values (outliers) for each of the 16 conditions and additionally one extra of the gender-emotion conditions with highest variance female-sadness, male-sadness, female-love, male-love, i.e. 52 values out of 512 data points in total.…”

Designing for a Wearable Affective Interface for the NAO Robot: A Study of Emotion Conveyance by Touch

“…Studies have typically shown that multiple dimensions of expression facilitate classification analyses [39,44]. We have demonstrated the potential for tactile interaction to provide much information regarding emotional conveyance; nevertheless, emotional interaction in naturalistic contexts is almost always of a context-specific (e.g., dependent on the type of task) and multi-sensory nature.…”

“…This is a key selling point of adopting textile wearables for use in affective-based human-robot tactile interaction. Studies indicate that emotion decoding is more accurate and more typical, when multi-modal sensors that pick up specific emotional and gestural information are added ( [39], also see [44]). Such multi-modal encoding and decoding allows for contextual nuance in the affective interaction.…”

Deep Facial Expression Recognition: A Survey

“…Compared with RNN, CNN is more suitable for computer vision applications; hence, its derivative C3D [107], which uses 3D convolutional kernels with shared weights along the time axis instead of the traditional 2D kernels, has been widely used for dynamic-based FER (e.g., [83], [108], [189], [197], [198]) to capture the spatio-temporal features. Based on C3D, many derived structures have been designed for FER.…”