Prosody in the Auditory and Visual Domains: A Developmental Perspective

Abstract

The development of body movements such as hand or head gestures, or facial expressions, seems to go hand-in-hand with the development of speech abilities. We know that very young infants rely on the movements of their caregivers' mouth to segment the speech stream, that infants' canonical babbling is temporally related to rhythmic hand movements, that narrative abilities emerge at a similar time in speech and gestures, and that children make use of both modalities to access complex pragmatic intentions. Prosody has emerged as a key linguistic component in this speech-gesture relationship, yet its exact role in the development of multimodal communication is still not well understood. For example, it is not clear what the relative weights of speech prosody and body gestures are in language acquisition, or whether both modalities develop at the same time or whether one modality needs to be in place for the other to emerge. The present paper reviews existing literature on the interactions between speech prosody and body movements from a developmental perspective in order to shed some light on these issues.

Entrainment and Modulation of Gesture–Speech Synchrony Under Delayed Auditory Feedback

“…() have compared different prosodic properties as they relate with gesture, and they found preliminary evidence that gesture is more likely coordinated with F0 modulations (rather than, e.g., articulatory lengthening). Our current method of taking peak F0 as an anchor, further provides an objective automatable method suitable for non‐scripted speech that replaces otherwise laborious inter‐rater judgments (ToBi) and context specific measures (e.g., vowel‐to‐vowel midpoint is only relevant if there are two vowels) that are often employed in scripted and controlled contexts (Rusiewicz et al., ) or otherwise applied for small amount of data (e.g., Loehr, ; Shattuck‐Huffnagel, ). Finally, note that we will also corroborate our F0 analyses by using another speech property that we will relate to gesture, namely the amplitude envelope of speech (see below; Fig.…”

“…Although evidence for entrained gesture–speech rhythms is largely based on repetitive pointing‐, beat‐tapping, or finger‐tapping gestures, where speech and gesture are scripted and produced on command (but see Danner et al., ), the findings appear generalizable to more spontaneous and semantically rich gestures on the basis of careful (but subjective) analysis of video recordings (e.g., McClave, ; McNeill, , ; Loehr, , ; for an overview see Wagner, Malisz, & Kopp, ). Given this growing evidence for the entrainment of gesture–speech rhythms, the question arises of how and why gestures are so closely controlled with respect to the rhythm of speech (Esteve‐Gibert & Guellaï, ; Iverson & Thelen, ; Rusiewicz, ; Rusiewicz & Esteve‐Gibert, ; Wagner et al., ).…”

Gesture-speech physics in fluent speech and rhythmic upper limb movements

“…Mainstream understanding of the gesture-prosody link holds that it is not "biologically mandated" (p. 69. in 9 ; 29 ), requiring neural-cognitive timing mechanisms 30,31 that appear only after about 16 months of age 32 (see also 33 ). Recent work, however, has investigated a potential physical coupling of arm movements with speech via myofascial-tissue biomechanics.…”

The physical basis of gesture-speech synchrony: Exploratory study and pre-registration

“…Explanations of why gesture and speech synchronize are varied (Wagner, Malisz, & Kopp, 2014;Esteve-Gibert & Guellaï, 2018). They include arguments relating to communicative functions, such that the meaning of gesture and speech is less ambiguous (and optimally effective) when performed in synchrony (e.g., Krauss, 2000).…”