Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli

Abstract

During public addresses, speakers accompany their discourse with spontaneous hand gestures (beats) that are tightly synchronized with the prosodic contour of the discourse. It has been proposed that speech and beat gestures originate from a common underlying linguistic process whereby both speech prosody and beats serve to emphasize relevant information. We hypothesized that breaking the consistency between beats and prosody by temporal desynchronization, would modulate activity of brain areas sensitive to speech-gesture integration. To this aim, we measured BOLD responses as participants watched a natural discourse where the speaker used beat gestures. In order to identify brain areas specifically involved in processing hand gestures with communicative intention, beat synchrony was evaluated against arbitrary visual cues bearing equivalent rhythmic and spatial properties as the gestures. Our results revealed that left MTG and IFG were specifically sensitive to speech synchronized with beats, compared to the arbitrary vision-speech pairing. Our results suggest that listeners confer beats a function of visual prosody, complementary to the prosodic structure of speech. We conclude that the emphasizing function of beat gestures in speech perception is instantiated through a specialized brain network sensitive to the communicative intent conveyed by a speaker with his/her hands.

Beat Gestures and Syntactic Parsing: An ERP Study

“…Related to this issue, in an fMRI study, Biau et al. () showed that BOLD responses in the left middle temporal gyrus were sensitive to the temporal misalignment between beats and auditory speech, over and above when the same speech fragment was synchronized with a circle following the original (but not present) hand trajectories.…”

“…In any case, it would be relevant to settle the issue empirically by comparing N100/P200 modulations of word ERPs from beat gestures and visual cues without communicative intent in future investigations. Related to this issue, in an fMRI study, Biau et al (2016) showed that BOLD responses in the left middle temporal gyrus were sensitive to the temporal misalignment between beats and auditory speech, over and above when the same speech fragment was synchronized with a circle following the original (but not present) hand trajectories.…”

“…This study focuses on the listener's neural correlates expressing the interaction between the sight of the speaker's rhythmic hand gestures, commonly called "beat gestures" (McNeill, 1992), and the processing of the corresponding verbal utterance. Beats are rapid flicks of the hand that do not necessarily carry semantic information and are considered as a visual support to prosody (Balaguer, & Soto-Faraco, 2015;Biau, Morís Fernández, Holle, Avila, & Soto-Faraco, 2016;Biau, Torralba, Fuentemilla, de Diego Holle et al, 2012). Compared to other types of speech gestures, beats are by far the most frequent in conversations.…”

Spatial attention enhances cortical tracking of quasi-rhythmic visual stimuli

“…Recent research has therefore acknowledged the importance of temporal structure that carries only limited temporal regularity and can thus be considered quasi-rhythmic. In particular, brain activity couples to quasi-rhythmic natural stimulation during lip-reading and parsing hand gestures (Biau, Morís Fernández, Holle, Avila, & Soto-Faraco, 2016;Hauswald, Lithari, Collignon, Leonardelli, & Weisz, 2018;Park, Kayser, Thut, & Gross, 2016). The relevance of quasirhythmicity for visual perception has also been shown in monkey single cell recordings: Neurons in area MT and extrastriate cortex better discriminate concurrent quasi-rhythmic than constant-motion stimuli because they capitalise on the temporal fine structure of the visual input (Burac̆as, Zador, DeWeese, & Albright, 1998).…”

“…Electrical source imaging has implicated V1 consistently (Andersen & Muller, 2010;Keil et al, 2012;Keitel, Andersen, Quigley, & Müller, 2013, but see Hillyard et al, 1997 and other visual cortices, such as V4, the lateral occipital complex (LOC) and human area MT have been screened for attentional modulation after being pre-selected as regions of interest (Lauritzen, Ales, & Wade, 2010;Palomares, Ales, Wade, Cottereau, & Norcia, 2012). Studies investigating the cortical processing of naturally occurring quasi-rhythmic visual stimuli in low frequency bands (< 7 Hz), such as the tracking of a speaker's lips movements (Hauswald, Lithari, Collignon, Leonardelli, & Weisz, 2018;Park, Kayser, Thut, & Gross, 2016) or hand gestures (Biau, Morís Fernández, Holle, Avila, & Soto-Faraco, 2016), have localized sources in circumscribed visual cortices without looking into detailed mapping of cortical regions and a modulation of the tracking response by visuo-spatial attention along the visual hierarchy.…”