Mimebot—Investigating the Expressibility of Non-Verbal Communication Across Agent Embodiments

Abstract

Unlike their human counterparts, arti cial agents such as robots and game characters may be deployed with a large variety of face and body con gurations. Some have articulated bodies but lack facial features, and others may be talking heads ending at the neck. Generally, they have many fewer degrees of freedom than humans through which they must express themselves, and there will inevitably be a ltering e ect when mapping human motion onto the agent. In this paper, we investigate ltering e ects on three types of embodiments, a) an agent with a body but no facial features, b) an agent with a head only and c) an agent with a body and a face. We performed a full performance capture of a mime actor enacting short interactions varying the non-verbal expression along ve dimensions (e.g. level of frustration and level of certainty) for each of the three embodiments. We performed a crowd sourced evaluation experiment comparing the video of the actor to the video of an animated robot for the di erent embodiments and dimensions. Our ndings suggest that the face is especially important to pinpoint emotional reactions, but is also most volatile to ltering e ects. e body motion on the other hand had more diverse interpretations, but tended to preserve the interpretation a er mapping, and thus proved to be more resilient to ltering.

A Comprehensive Review of Data‐Driven Co‐Speech Gesture Generation

“…Apart from the speaker identity, the emotional or affective state of a speaker also impacts the gestures performed by them. A striking example of this is the large range of expressive motion variation with the same lexical message explored in the Mimebot data [AONB17]. Building emotionally aware embodied agents is a common research direction [CBFV16, SZGK18].…”

Perceptual Evaluation of Blended Sonification of Mechanical Robot Sounds Produced by Emotionally Expressive Gestures: Augmenting Consequential Sounds to Improve Non-verbal Robot Communication

“…The frustrated sound file was 9 seconds long, the relaxed sound file was 6.5 seconds and the two joyful sound files were 10 versus 9 seconds. The two joyful sounds were produced by two gestures that differed in terms of variation of the non-verbal expression along a joyful axis, as described in [1]. Two versions were included for comparative purposes.…”

“…Two versions were included for comparative purposes. The gesture that produced the first sound file was rated as more joyful than the gesture producing the second sound (see [1]).…”

Real-time labeling of non-rigid motion capture marker sets

“…We also demonstrate how our method is used together with data-driven marker reconstruction to provide high quality data from sparse marker sets. For this experiment, we used a subset of the data from a previous study on expressive artificial agents captured in our motion capture lab [26] . The data consist of an 8.4 min (60,315 frames) long motion capture clip of an actor giving instructions to an interlocutor, while varying the displayed level of engagement from very un-engaged to very engaged, as well as two shorter clips of RoM data (one for the hands/fingers and one for the face).…”