Towards integrated microplanning of language and iconic gesture for multimodal output

Abstract

When talking about spatial domains, humans frequently accompany their explanations with iconic gestures to depict what they are referring to. For example, when giving directions, it is common to see people making gestures that indicate the shape of buildings, or outline a route to be taken by the listener, and these gestures are essential to the understanding of the directions. Based on results from an ongoing study on language and gesture in direction-giving, we propose a framework to analyze such gestural images into semantic units (image description features), and to link these units to morphological features (hand shape, trajectory, etc.). This feature-based framework allows us to generate novel iconic gestures for embodied conversational agents, without drawing on a lexicon of canned gestures. We present an integrated microplanner that derives the form of both coordinated natural language and iconic gesture directly from given communicative goals, and serves as input to the speech and gesture realization engine in our NUMACK project.

Pursuing and demonstrating understanding in dialogue

“…SmartKom [Wahlster et al, 2001] is an influential early example of multimodal fission for dialogue generation. By contrast, problem-solving models of multimodal generation, such as Cassell et al [2000], Kopp et al [2004], reason about the affordances and interdependencies of body and speech to creatively explore the space of possible multimodal utterances and synthesize utterances that link specific behaviors to specific functions opportunistically and flexibly.…”

Data-based analysis of speech and gesture: the Bielefeld Speech and Gesture Alignment corpus (SaGA) and its applications

“…A systematic extension that is based on the anatomically joints of arm and hands that bring about movement has been implemented in the FORM scheme [47]. The same basic approach has also been pursued by Kopp et al [39]. The kinematic-or "morphologic", as we call it-part of the SaGA scheme is closely related to the latter two schemes.…”

“…The scheme of Calbris is only concerned with "straight-line gestures in space […]" [15, p. 104] but not with gestural movements of any kind as we are here. The present annotation schemes builds on the scheme used in Kopp et al [39] to capture two-handed gestures and the manifold configurations they can manifest.…”

“…However, Johnston's gesture representations are rather impoverished, they are strictly limited by the restricted pen-input application domain. In the later work of Kopp et al [39], attribute-value pairs for representing a gesture are oriented at the "morphology" of gestures. The architecture of such gesture AVMs is straightforward: the orientations and movements of palm, back of hand and wrist make up features that are specified by appropriate orientation and movement values (including the empty value for, say, static gestures).…”

Systematicity and Idiosyncrasy in Iconic Gesture Use: Empirical Analysis and Computational Modeling

“…Section 2 introduces the experimental setting and the data coding methodology, Section 3 presents results from the corpus analysis. Based on these findings, we describe in Section 4 a computational modeling account that goes beyond previous systems, which either rely on generalized rule-based models that disregard idiosyncrasy in gesture use [6,18], or employ data-based methods that approximate single speakers but have difficulties with extracting systematicities of gesture use. These data-based approaches are typically (and successfully) employed to generate gesturing behavior which has no particular meaning-carrying function, e.g., discourse gestures [27] or beat gestures (Theune & Brandhorst, this volume).…”