Speaker's hand gestures modulate speech perception through phase resetting of ongoing neural oscillations

Abstract

Speakers often accompany speech with spontaneous beat gestures in natural spoken communication. These gestures are usually aligned with lexical stress and can modulate the saliency of their affiliate words. Here we addressed the consequences of beat gestures on the neural correlates of speech perception. Previous studies have highlighted the role played by theta oscillations in temporal prediction of speech. We hypothesized that the sight of beat gestures may influence ongoing low-frequency neural oscillations around the onset of the corresponding words. Electroencephalographic (EEG) recordings were acquired while participants watched a continuous, naturally recorded discourse. The phase-locking value (PLV) at word onset was calculated from the EEG from pairs of identical words that had been pronounced with and without a concurrent beat gesture in the discourse. We observed an increase in PLV in the 5-6 Hz theta range as well as a desynchronization in the 8-10 Hz alpha band around the onset of words preceded by a beat gesture. These findings suggest that beats help tune low-frequency oscillatory activity at relevant moments during natural speech perception, providing a new insight of how speech and paralinguistic information are integrated.

Beat Gestures and Syntactic Parsing: An ERP Study

“…Beats may modulate the visual context in a nonrandom manner, preparing the system for upcoming auditory inputs and having an effect on how they are processed. In line with this assumption, we previously showed that beat gestures induced the synchronization in the listeners’ EEG in the theta band at (and even before) the onset of the affiliate word (Biau & Soto‐Faraco, ; Biau et al., ), suggesting an anticipatory effect of gestures on the processing of the sensory input at relevant moments in the auditory signal (Astheimer & Sanders, ). Cross‐modal anticipation in audiovisual speech has been reported using several approaches, including ERPs (Brunellière & Soto‐Faraco, ; Brunellière & Soto‐Faraco, ; van Wassenhove et al., ) and behavior (Sánchez‐García, Alsius, Enns, & Soto‐Faraco, ), and seems to be one of the ways in which audiovisual integration confers a benefit in speech processing.…”

“…This study focuses on the listener's neural correlates expressing the interaction between the sight of the speaker's rhythmic hand gestures, commonly called "beat gestures" (McNeill, 1992), and the processing of the corresponding verbal utterance. Beats are rapid flicks of the hand that do not necessarily carry semantic information and are considered as a visual support to prosody (Balaguer, & Soto-Faraco, 2015;Biau, Morís Fernández, Holle, Avila, & Soto-Faraco, 2016;Biau, Torralba, Fuentemilla, de Diego Holle et al, 2012). Compared to other types of speech gestures, beats are by far the most frequent in conversations.…”

Prediction and constraint in audiovisual speech perception

“…The authors indeed found more consistent neural phase relationships in auditory cortex in response to the same acoustic input when the visual input matched the auditory signal compared to when it did not. These results suggest that congruent visual information can increase the precision of oscillatory auditory activity in the 4–8 Hz range (for an example with gesture, see Biau, Torralba, Fuentemilla, de Diego Balaguer, & Soto-Faraco, In press). …”

Dynamic Predictions: Oscillatory Mechanisms Underlying Multisensory Sequence Processing

“…Arnal et al (2011) investigated violation of multisensory predictions in speech perception and demonstrated that correct crossmodal predictions were associated with delta oscillations, whereas prediction errors were predominantly accompanied by beta-and gammaband activity. Another recent study on crossmodal effects in speech perception (Biau et al 2015) has observed a modulation of theta-oscillations during auditory speech processing by concomitant visual input from observation of gestures. Low-frequency oscillations have also been found to be associated with temporal prediction in a study where a visual color change predicted the occurrence of an auditory stimulus (van Wassenhove and Grzeczkowski 2015).…”