How Referential Gestures Align With Speech: Evidence From Monolingual and Bilingual Speakers

Graziano, Maria; Nicoladis, Elena; Marentette, Paula

doi:10.1111/lang.12376

Cited by 17 publications

(12 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…While there are several theories on how gestures are produced by humans [5,10,33], there is a consensus that speech and gestures correlate strongly [18,23,31,39]. In this section, we review some concepts relevant to our work, namely gesture classification, the temporal alignment between gestures and speech as well as the gesture-generation problem formulation.…”

Section: Background and Related Work 21 Backgroundmentioning

confidence: 99%

See 1 more Smart Citation

Gesticulator: A framework for semantically-aware speech-driven gesture generation

Kucherenko

Jonell

Waveren

et al. 2020

Proceedings of the 2020 International Conference on Multimodal Interaction

130

120

View full text Add to dashboard Cite

During speech, people spontaneously gesticulate, which plays a key role in conveying information. Similarly, realistic co-speech gestures are crucial to enable natural and smooth interactions with social agents. Current end-to-end co-speech gesture generation systems use a single modality for representing speech: either audio or text. These systems are therefore confined to producing either acoustically-linked beat gestures or semantically-linked gesticulation (e.g., raising a hand when saying "high"): they cannot appropriately learn to generate both gesture types. We present a model designed to produce arbitrary beat and semantic gestures together. Our deep-learning based model takes both acoustic and semantic representations of speech as input, and generates gestures as a sequence of joint angle rotations as output. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach. The code and video are available at the project page svito-zar.github.io/gesticulator.

show abstract

Section: Background and Related Work 21 Backgroundmentioning

confidence: 99%

“…2.1.2 Gesture-Speech Alignment. Gesture-speech alignment is an active research field covering several languages, including French [12], German [4], and English [18,31,39]. We focus on prior work on gesture-speech alignment for the English language.…”

Section: Background and Related Work 21 Backgroundmentioning

confidence: 99%

Gesticulator: A framework for semantically-aware speech-driven gesture generation

Kucherenko

Jonell

Waveren

et al. 2020

Proceedings of the 2020 International Conference on Multimodal Interaction

130

120

View full text Add to dashboard Cite

show abstract

“…It is widely accepted that the onset of a whole gesture phrase most often comes before the lexical affiliate (Schegloff, 1984;Morrel-Samuels and Krauss, 1992;Hadar and Butterworth, 1997;McNeill, 2005;Church et al, 2014;ter Bekke et al, 2020) and occurs during speech not pauses (Nobe, 2000;Chui, 2005), but there are inconsistent findings concerning the gesture stroke. Several studies of various languages have demonstrated that the stroke onset starts (and sometimes even ends) before the affiliate (Schegloff, 1984;Ferré, 2010;Bergmann et al, 2011;ter Bekke et al, 2020) while several others report that it is synchronized with the affiliate (Chui, 2005;Graziano et al, 2020) or the co-expressed speech (McNeill, 2005), which automatically implies that the gesture onset is prepositioned anyway. What many of these studies additionally show is that strokes produced after the affiliate turn out to be rare.…”

Section: Gesture Positionmentioning

confidence: 99%

Action-Depicting Gestures and Morphosyntax: The Function of Gesture-Speech Alignment in the Conversational Turn

Urbanik

Svennevig

2021

Front. Psychol.

View full text Add to dashboard Cite

The current study examines the role of action-depicting gestures in conversational turns by focusing on their semantic characteristics and temporal position in relation to their verbal affiliates (action verbs or more complex verb phrases). The data are video recordings of naturally occurring interactions in multilingual construction sites in Norway. The analysis distinguishes two modes of action depiction: generic depictions, which represent the action as a general type, and contextualized depictions, which in addition include deictic references to the spatio-material environment or iconic representations of the specific manner of action performance. These two modes typically occupy different positions in the turn. Generic depictions are mostly initiated before the verbalization of the action or are synchronized with it, while contextualized depictions mostly start simultaneously with the verbalization and extend beyond the verb phrase or the turn. The pre-positioned and synchronized generic gestures are shown to serve as a practice for facilitating recognition of the verbalized action and may be temporally manipulated in order to pre-empt understanding problems in the face of reduced common linguistic resources. The post-positioned contextualized depictions serve instead to add specifying information about aspects of the action referred to and thereby to complement or supplement the meaning of the verb phrase, securing understanding of action specifics. The study contributes to research on gesture-speech synchrony by demonstrating how variation in the alignment of action depiction and syntax is used to direct the recipient’s attention toward different interactional goals.

show abstract

“…Hence, they may play a facilitative role in L2 lexical stress acquisition. For instance, the temporal relation between referential gestures and the co-occurring speech is the same for monolingual and bilingual speakers (Graziano et al, 2020). Presumably this finding also holds for beat gestures, which can be argued to be even more closely related to speech temporally (Wagner et al, 2014).…”

Section: Gesture and L2 Acquisitionmentioning

confidence: 87%

Gesture-speech coupling in L2 lexical stress production: A pre-registration of a speech acoustic and gesture kinematic study

Bosker¹,

Hoetjes²,

Pouw³

et al. 2021

Preprint

View full text Add to dashboard Cite

The prosody of a second language (L2) is notoriously difficult to acquire. It requires the mastery of a range of nested multimodal systems, including articulatory but also gestural signals, as hand gestures are produced in close synchrony with spoken prosody. It remains unclear how easily the articulatory and gestural systems acquire new prosodic patterns in the L2 and how the two systems interact, especially when L1 patterns interfere. This interdisciplinary pre-registered study investigates how Dutch learners of Spanish produce multimodal lexical stress in Spanish-Dutch cognates (e.g., Spanish profeSOR vs. Dutch proFESsor). Acoustic analyses assess whether gesturing helps L2 speakers to place stress on the correct syllable; and whether gesturing boosts the acoustic correlates of stress through biomechanic coupling. Moreover, motion-tracking and time-series analyses test whether gesture-prosody synchrony is enhanced for stress-matching vs. stress-mismatching cognate pairs, perhaps revealing that gestural timing is biased in the L1 (or L2) direction (e.g., Spanish profeSOR with the gesture biased towards Dutch stressed syllable -fes). Thus, we will uncover how speakers deal with manual, articulatory, and cognitive constraints that need to be brought in harmony for efficient speech production, bearing implications for theories on gesture-speech interaction and multimodal L2 acquisition.

show abstract

How Referential Gestures Align With Speech: Evidence From Monolingual and Bilingual Speakers

Cited by 17 publications

References 56 publications

Gesticulator: A framework for semantically-aware speech-driven gesture generation

Gesticulator: A framework for semantically-aware speech-driven gesture generation

Action-Depicting Gestures and Morphosyntax: The Function of Gesture-Speech Alignment in the Conversational Turn

Gesture-speech coupling in L2 lexical stress production: A pre-registration of a speech acoustic and gesture kinematic study

Contact Info

Product

Resources

About