The phase of cortical oscillations determines the perceptual fate of visual cues in naturalistic audiovisual speech

Abstract

When we see our interlocutor, our brain seamlessly extracts visual cues from their face and processes them along with the sound of their voice, making speech an intrinsically multimodal signal. Visual cues are especially important in noisy environments, when the auditory signal is less reliable. Neuronal oscillations might be involved in the cortical processing of audiovisual speech by selecting which sensory channel contributes more to perception. To test this, we designed computer-generated naturalistic audiovisual speech stimuli where one mismatched phoneme-viseme pair in a key word of sentences created bistable perception. Neurophysiological recordings (high-density scalp and intracranial electroencephalography) revealed that the precise phase angle of theta-band oscillations in posterior temporal and occipital cortex of the right hemisphere was crucial to select whether the auditory or the visual speech cue drove perception. We demonstrate that the phase of cortical oscillations acts as an instrument for sensory selection in audiovisual speech processing.

Oscillatory tracking of pseudo-rhythmic speech is constrained by linguistic predictions

“…Strikingly, when the pseudo-rhythmicity in speech matches the predictions of the internal model, responses were more rhythmic for matched pseudo-rhythmic compared to isochronous speech input. Our model is optimally sensitive to natural speech variations, can explain phase dependent speech categorization behavior [31, 35, 44, 63], and naturally comprises a neural phase code [40, 42, 43]. These results show that part of the pseudo-rhythmicity of speech is expected by the brain and it is even optimized to process it in this manner, but only when it follows the internal model.…”

“…In the domain of working memory, this type of phase precession has been shown in rat hippocampus [28, 29] and more recently in human electroencephalography [30]. In speech, phase of activation and perceived content are also associated [31-35] and phase has been implicated in tracking of higher-level linguistic structure [18, 36, 37]. However, the direct link between phase and the predictability flowing from a language model has yet to be established.…”

An oscillating computational model can track pseudo-rhythmic speech by using linguistic predictions

“…Strikingly, when the pseudo-rhythmicity in speech matches the predictions of the internal model, responses were more rhythmic for matched pseudo-rhythmic compared to isochronous speech input. Our model is optimally sensitive to natural speech variations, can explain phase-dependent speech categorization behavior ( Ten Oever and Sack, 2015 ; Thézé et al, 2020 ; Reinisch and Sjerps, 2013 ; Ten Oever et al, 2020 ), and naturally comprises a neural phase code ( Panzeri et al, 2015 ; Mehta et al, 2002 ; Lisman and Jensen, 2013 ). These results show that part of the pseudo-rhythmicity of speech is expected by the brain and it is even optimized to process it in this manner, but only when it follows the internal model.…”

“…In the domain of working memory, this type of phase precession has been shown in rat hippocampus ( O'Keefe and Recce, 1993 ; Malhotra et al, 2012 ) and more recently in human electroencephalography ( Bahramisharif et al, 2018 ). In speech, phase of activation and perceived content are also associated ( Ten Oever and Sack, 2015 ; Kayser et al, 2016 ; Di Liberto et al, 2015 ; Ten Oever et al, 2016 ; Thézé et al, 2020 ), and phase has been implicated in tracking of higher-level linguistic structure ( Meyer, 2018 ; Brennan and Martin, 2020 ; Kaufeld et al, 2020a ). However, the direct link between phase and the predictability flowing from a language model has yet to be established.…”

Audiovisual Interactions Among Near-Threshold Oscillating Stimuli in the Far Periphery Are Phase-Dependent

“…However, the current findings can be put into perspective based on existing evidence of mechanisms that underlie cross-sensory effects. The oscillatory phase of the internal cortical rhythms is known to play a role in auditory perception through interactions with cross-sensory visual and motor cues ( Mercier et al, 2015 ; Simon and Wallace, 2017 ; Benedetto et al, 2018 ; Ikumi et al, 2019 ; Bauer et al, 2020 ; Mégevand et al, 2020 ; Thézé et al, 2020 ; Assaneo et al, 2021 ). There is even evidence of “oscillatory phase-resetting” in multisensory interactions among early sensory cortices ( Lakatos et al, 2007 ; Doesburg et al, 2008 ; Schroeder and Lakatos, 2009 ; Atilgan et al, 2018 ).…”