Modeling the Interaction between Speech and Gesture.

Cassell, Justine; Stone, Matthew; Douville, Brett; Prevost, Scott; Achorn, Brett

doi:10.21236/ada290549

Cited by 44 publications

(18 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This latter was extracted at regular time steps of 10 ms with Praat [3], a package including the pitch extraction technique described in [2]. The motivation behind this choice is not only that silence segments are characterized by frequencies way higher than those observed in speech, but also that the pitch tends to be correlated with the "beat" gesture typically accompanying syllables where the intonation is stressed [4,20]. Then, in order to synchronize audio and video data, audio was resampled according to the video frame rate, averaging the pitch values occur-ring in each time period.…”

Section: Methodsmentioning

confidence: 99%

Look at Who’s Talking: Voice Activity Detection by Automated Gesture Analysis

Cristani

Pesarin

Vinciarelli

et al. 2012

Communications in Computer and Information Science

View full text Add to dashboard Cite

Abstract. This paper proposes an approach for Voice Activity Detection (VAD) based on the automatic measurement of gesturing. The main motivation of the work is that gestures have been shown to be tightly correlated with speech, hence they can be considered a reliable evidence that a person is talking. The use of gestures rather than speech for performing VAD can be helpful in many situation (e.g., surveillance and monitoring in public spaces) where speech cannot be obtained for technical, legal or ethical issues. The results show that the gesturing measurement approach proposed in this work achieves, on a frame-by-frame basis, an accuracy of 71 percent in distinguishing between speech and non-speech.

show abstract

Section: Methodsmentioning

confidence: 99%

Look at Who’s Talking: Voice Activity Detection by Automated Gesture Analysis

Cristani

Pesarin

Vinciarelli

et al. 2012

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…The present work is inter alia motivated by the proposal of Cassell et al [1], who suggested creating predictive models for embodied conversational agents (ECAs) to test theories on speech and gesture interaction. The results presented here contribute to a data-driven model for MDEs.…”

Section: Introductionmentioning

confidence: 99%

Interaction between Speech and Gesture: Strategies for Pointing to Distant Objects

Pfeiffer

2012

Gesture and Sign Language in Human-Computer Interaction and Embodied Communication

View full text Add to dashboard Cite

Abstract. Referring to objects using multimodal deictic expressions is an important form of communication. This work addresses the question on how content is distributed between the modalities speech and gesture by comparing deictic pointing gestures to objects with and without speech. As a result, two main strategies used by participants to adapt their gestures to the condition were identified. This knowledge can be used, e.g., to improve the naturalness of pointing gestures employed by embodied conversational agents.

show abstract

“…While an expert animator may realize this unconsciously in the "look" of a properly animated character, a program to automatically generate motions must embed this knowledge in a system of rules. A fuller report of some aspects of this work appears in [9] and [10].…”

Section: Introductionmentioning

confidence: 99%

Synthesizing cooperative conversation

Pélachaud

Cassell

Badler

et al. 1998

Multimodal Human-Computer Communication

Self Cite

View full text Add to dashboard Cite

We describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversations are created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gesture generators. Disciplines AbstractWe describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversations are created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gesture generators.

show abstract

Modeling the Interaction between Speech and Gesture.

Cited by 44 publications

References 17 publications

Look at Who’s Talking: Voice Activity Detection by Automated Gesture Analysis

Look at Who’s Talking: Voice Activity Detection by Automated Gesture Analysis

Interaction between Speech and Gesture: Strategies for Pointing to Distant Objects

Synthesizing cooperative conversation

Contact Info

Product

Resources

About