Modeling the Interaction between Speech and Gesture

Abstract

Until now theories of the gesture-speech relationship have been difficult to evaluate because of their descriptive basis. In this paper we provide a tool for investigating the relationship between speech and gesture: a system that generates speech, intonation, and gesture using two copies of an identical program that have different knowledge of the world and must cooperate to accomplish a goal. The output of the dialogue generation is fed into a three-dimensional interactive animated model-two graphic figures on a computer screen who gesture according to the rules given to the system. The advantage of computer modeling in this domain is that it forces us to come up with predictive theories of the gesture-speech relationship. A felicitous outcome is a working system to realize autonomous animated conversational agents, for virtual reality and other purposes.

Gesture Recognition

“…• Iconic -Representational gestures depicting some feature of the object, action, or event being described • Metaphoric -Gestures that represent a common metaphor, rather than the object or event directly • Beat -Small, formless gestures, often associated with word emphasis • Deictic -Pointing gestures that refer to people, objects, or events in space or time These gesture types modify the content of accompanying speech and often help to disambiguate speech, similar to the role of spoken intonation. Cassell et al [6] described early research in conversational agents that models the relationship between speech and gesture and generates interactive dialogs between threedimensional animated characters that gesture as they speak.…”

Gesture, Prosody and Information Structure Synchronisation in Turkish

“…Although McNeill (1992) and McNeill and Duncan (2000) were not interested in information structure per se, their claim constitutes a possible justification for the existence of an association between information structure (through focus) and gesture. Another similar presumption was made in Cassell et al (1994). The study was interested in 3D animation, and they selected focal areas of utterances as the domain where gestures would be anchored, based on the view that gestures are informative and can be used in cooperation with speech to carry discourse forward.…”

Η Πολυτροπικότητα Σε Τηλεοπτικές Συνομιλίες

“…Προς την κατεύθυνση αυτή έχουν υπάρξει πολλές προσπάθειες δημιουργίας εικονικών διαλογικών πρακτόρων, δηλ. διεπαφών στις οποίες το υπολογιστικό σύστημα αναπαριστάται ως άνθρωπος και η πληροφορία μεταδίδεται στους χρήστες μέσω διαφορετικών τροπικοτήτων, όπως η φωνή και οι χειρονομίες (Cassell et al, 1994;Kopp et al, 2004). Οι διαλογικοί πράκτορες μοιάζουν με ανθρώπους (π.χ.…”