Effects of perturbation and prosody on the coordination of speech and gesture

Entrainment and Modulation of Gesture–Speech Synchrony Under Delayed Auditory Feedback

“…In the DAF condition (half of the trials), participants heard their own speech with a 200 ms delay, which resulted in elongated spoken responses. In contrast to McNeill's (1992) original DAF study, increased gesture-speech asynchrony was found for DAF vs. NO DAF by Rusiewicz et al (2014). That is, the time difference of the gesture-launch midpoint and the vowel-to-vowel midpoint of the referenced target word increased under DAF.…”

“…Importantly, McNeill's (1992) original descriptive report does suggest that gesture-speech synchrony is maintained under DAF, even when it concerns spontaneous gestures produced during narration. However, as promising as these results are, more empirical detail is required to fully understand how gesture-speech synchrony is affected by DAF for spontaneous gesturing (as also argued by Rusiewicz et al, 2014). As such, we aim to conceptually replicate McNeill's (1992) classic setup by assessing the effect of DAF during narration of a cartoon while objectively tracking movement and speech with high resolution.…”

“…In addition to the issue of whether gesture is reactive to speech perturbation, we aim to address the question how and why it is reactive. It has been argued from a dynamical systems perspective of gesture (McNeill, 1992(McNeill, , 2005Rusiewicz et al, 2014; see also Rusiewicz et al, 2011) that the stability of gesture-speech synchrony under DAF suggests that gesture and speech flexibly organize into equivalent functional sensorimotor solutions through tight bidirectional coupling of the systems. This stability is likened to the classic finding by Kelso et al (1984), who showed that when the jaw is locked in the midst of articulating a syllable, the lower lip or tongue spontaneously jumps into place to successfully complete the syllable (depending whether the perturbation is successfully resolved by a lip or tongue intervention).…”

“…Yet it has also been shown that gesture-speech synchrony remains relatively stable under perturbations of hand movements or speech production, suggesting continuous bidirectional coupling of gesture and speech (Chu & Hagoort, 2014;McNeill, 1992;Rusiewicz et al, 2014). Chu and Hagoort (2014) found that when visual feedback of a pointing gesture is disrupted, speech will halt so as to synchronize with the perturbed (and therefore delayed) pointing movement.…”

“…Detailed investigation of the kinematics of gesticulation under DAF obtained mixed results, however. Rusiewicz et al (2014) used a task in which participants pointed and verbally labeled targets. In the DAF condition (half of the trials), participants heard their own speech with a 200 ms delay, which resulted in elongated spoken responses.…”