Multimodal Integration of Emotional Signals from Voice, Body, and Context: Effects of (In)Congruence on Emotion Recognition and Attitudes Towards Robots

Abstract

Humanoid social robots have an increasingly prominent place in today's world. Their acceptance in social and emotional human-robot interaction (HRI) scenarios depends on their ability to convey well recognized and believable emotional expressions to their human users. In this article, we incorporate recent findings from psychology, neuroscience, human-computer interaction, and HRI, to examine how people recognize and respond to emotions displayed by the body and voice of humanoid robots, with a particular emphasis on the effects of incongruence. In a social HRI laboratory experiment, we investigated contextual incongruence (i.e., the conflict situation where a robot's reaction is incongrous with the socio-emotional context of the interaction) and cross-modal incongruence (i.e., the conflict situation where an observer receives incongruous emotional information across the auditory (vocal prosody) and visual (whole-body expressions) modalities). Results showed that both contextual incongruence and cross-modal incongruence confused observers and decreased the likelihood that they accurately recognized the emotional expressions of the robot. This, in turn, gives the impression that the robot is unintelligent or unable to express "empathic" behaviour and leads to profoundly harmful effects on likability and believability. Our findings reinforce the need of proper design of emotional expressions for robots that use several channels to communicate their emotional states in a clear and effective way. We offer recommendations regarding design choices and discuss future research areas in the direction of multimodal HRI.

“Grumpy” or “furious”? arousal of emotion labels influences judgments of facial expressions

“…Similarly, future research should investigate the influence of faces on label ratings, to gain a more complete understanding of the potential bidirectional relationship between language and emotional expressions. We expect that label ratings would be influenced by paired face information; similar bidirectional relationships have previously been shown between face-body and face-voice pairings [74][75][76][77][78][79][80], and we believe this would hold for face-language pairings.…”

On Designing Expressive Robot Behavior: The Effect of Affective Cues on Interaction

“…The robot behavioral multimodality refers to coordinating and combining different modalities of communication in the robot (agent) behavior, which has been a challenging research topic over the last years [38,84]. In facial expressions and gestures coordination, among others, Clavel et al [23] discussed the positive effect of facial and bodily expressions on the affective expressivity of a virtual character (and consequently emotion recognition), and Costa et al [25] proved that gestures can effectively help in recognizing the facial expressions of a robot.…”

Multi-Modality Emotion Recognition Model with GAT-Based Multi-Head Inter-Modality Attention

“…Naturally, the emotion perception of humans is not just determined by one type of information; it is triggered by a multitude of factors or signals emitted from others. Many studies have utilized multimodality (i.e., visual, audio, and text) to improve the performance of emotion recognition [ 19 , 21 , 22 , 23 , 51 ]. Zhou et al [ 51 ] and Tripathi et al [ 19 ] modeled the relationships among text, visual, and audio modalities by deep learning methods to improve performance.…”

“…Naturally, emotion perception by humans is not decided by just one type of information; it is triggered by a multitude of factors or signals emitted from others. By investigating such factors, many researchers have proposed multi-modality approaches to improve the performance of emotion recognition [ 21 , 22 , 23 , 24 , 25 , 26 ].…”