Modeling Embodied Feedback with Virtual Humans

Abstract

Abstract. In natural communication, both speakers and listeners are active most of the time. While a speaker contributes new information, a listener gives feedback by producing unobtrusive (usually short) vocal or non-vocal bodily expressions to indicate whether he/she is able and willing to communicate, perceive, and understand the information, and what emotions and attitudes are triggered by this information. The simulation of feedback behavior for artificial conversational agents poses big challenges such as the concurrent and integrated perception and production of multi-modal and multi-functional expressions. We present an approach on modeling feedback for and with virtual humans, based on an approach to study "embodied feedback" as a special case of a more general theoretical account of embodied communication. A realization of this approach with the virtual human Max is described and results are presented.

Co-constructing Grounded Symbols—Feedback and Incremental Adaptation in Human–Agent Dialogue

“…' Contrary to common belief, communicative feedback is not merely a way of signalling the interlocutor to continue speaking, but a powerful mechanism that enables listeners to express their mental state towards the speaker's utterance. According to Allwood and colleagues [1,19], communicative feedback signals express the basic communicative functions 1 'contact' (being "willing and able to continue the interaction"), 'perception' (being "willing and able to perceive the message"), 'understanding' (being "willing and able to understand the message"), and 'attitudinal reactions' (being "willing and able to react and (adequately) respond to the message") such as 'acceptance' or 'agreement' [1, p. 3]. These functions are related to each other hierarchically [1,11] such that higher functions imply lower functions (when signalling feedback of type understanding, for example, successful perception and contact are implied) and lower functions block higher functions (e.g., feedback of failed perception entails a problem in understanding).…”

“…Recently, the increase in the capabilities of incremental natural language understanding, has directed attention to the question of what type of feedback should be provided [19,24,23].…”

SceneMaker: Intelligent Multimodal Visualisation of Natural Language Scripts

“…These facts show the importance of the visualisation of body language in film/play production, but also point out the challenges in deriving information for animation from scripts containing mostly dialogues. Much research is dedicated to modelling of emotion and facial expressions, gaze and hand gestures (Kopp et al, 2008;Sowa, 2008), but body posture has yet to be addressed extensively (Gunes and Piccardi, 2006). The SceneMaker prototype will be developed using appropriate multimodal technology.…”

“…An example implementation can be found in the virtual human, Max (Kopp et al, 2008), who engages museum visitors in face-to-face small talk. Max listens while the users type their input, reasons about actions to take, has intentions and goal plans, reacts emotionally and gives verbal and non-verbal feedback.…”

“…The virtual agent systems, Improv (Perlin and Goldberg, 1996), AESOPWORLD (Okada et al, 1999), Behaviour Generation System (Breitfuss et al, 2007), Max (Kopp et al, 2008) and Greta (Pelachaud, 2005 andDe Rosis et al, 2003) investigate in great detail, how to model an agent's mind with sophisticated reasoning about emotional states, personality, social and conversational roles. Furthermore, these systems connect those mental processes to facial, gestural and vocal expressions of 3D models achieving human-like conversational behaviour.…”