Cortical tracking of formant modulations derived from silently presented lip movements and its decline with age

Abstract

The integration of visual and auditory cues is crucial for successful processing of speech, especially under adverse conditions. Recent reports have shown that when participants watch muted videos of speakers, the phonological information about the acoustic speech envelope, which is associated with but independent from the speakers’ lip movements, is tracked by the visual cortex. However, the speech signal also carries richer acoustic details, for example, about the fundamental frequency and the resonant frequencies, whose visuophonological transformation could aid speech processing. Here, we investigated the neural basis of the visuo-phonological transformation processes of these more fine-grained acoustic details and assessed how they change as a function of age. We recorded whole-head magnetoencephalographic (MEG) data while the participants watched silent normal (i.e., natural) and reversed videos of a speaker and paid attention to their lip movements. We found that the visual cortex is able to track the unheard natural modulations of resonant frequencies (or formants) and the pitch (or fundamental frequency) linked to lip movements. Importantly, only the processing of natural unheard formants decreases significantly with age in the visual and also in the cingulate cortex. This is not the case for the processing of the unheard speech envelope, the fundamental frequency, or the purely visual information carried by lip movements. These results show that unheard spectral fine details (along with the unheard acoustic envelope) are transformed from a mere visual to a phonological representation. Aging affects especially the ability to derive spectral dynamics at formant frequencies. As listening in noisy environments should capitalize on the ability to track spectral fine details, our results provide a novel focus on compensatory processes in such challenging situations.

Visual and auditory cortices represent acoustic speech-related information during silent lip reading

“…Previous work has shown that lip movements activate a network of temporal, parietal and frontal regions (Bourguignon et al, 2020; Calvert et al, 1997; Capek et al, 2008; O’Sullivan et al, 2017; Ozker et al, 2018; Paulesu et al, 2003; Pekkola et al, 2005) and that both occipital and motor regions can align their activity to the dynamics of lip movements (Park et al, 2018, 2016). The present data substantiate this, but also show that the representation of the physically visible lip trajectory along visual pathways is accompanied by the representation of spectral acoustic features, a type of selectivity not directly revealed previously (Suess et al, 2022). Spectral features are vital for a variety of listening tasks (Albouy et al, 2020; Bröhl and Kayser, 2020; Ding and Simon, 2013; Tivadar et al, 2020, 2018), and oro-facial movements provide concise information about the spectral domain.…”

“…Visual speech contains temporal information that can be predictive of subsequent acoustic signals and allows mapping visual cues onto phonological representations (Campbell, 2008;Lazard and Giraud, 2017). Importantly, the visual cortex tracks dynamic lip signals (Park et al, 2016) and, as suggested recently, may also restore the unheard acoustic envelope of visually presented speech (Hauswald et al, 2018;Suess et al, 2022). Importantly, the evidence that visual speech induces information about the unheard speech acoustics along both auditory and the visual pathways may not be mutually exclusive, as both may contribute to a supramodal frame of reference for speech (Arnal et al, 2009;Rauschecker, 2012).…”

“…These results advance our understanding of how the brain exploits lip movements for speech-related processes in a number of ways. The restoration of auditory features during silent lip reading had been suggested in two previous studies, one showing the coherence of temporal brain activity with the nonpresented speech envelope (Bourguignon et al, 2020) and another showing the coherence between occipital activity and the envelope (Hauswald et al, 2018;Suess et al, 2022). Yet, these studies differed in their precise experimental designs, their statistical procedures revealing the 'restoration' effect, and did not probe a direct link to comprehension performance.…”

MEG Activity in Visual and Auditory Cortices Represents Acoustic Speech-Related Information during Silent Lip Reading

“…One view is that the visual system directly contributes to establishing speech representations ( Bernstein et al, 2011 ; O’Sullivan et al, 2017 ; Ozker et al, 2018 ), as oro-facial movements provide temporal information that can be predictive of concurrent acoustic signals and allow mapping visual cues onto phonological representations ( Campbell, 2008 ; Lazard and Giraud, 2017 ). The visual cortex tracks dynamic lip signals ( Park et al, 2016 ) and, as suggested recently, may also directly “restore” the acoustic envelope of the visually presented speech ( Hauswald et al, 2018 ; Suess et al, 2022 ). Another view is that visual speech is mainly represented in regions of the auditory pathways, possibly exploiting speech-specific processes of this system.…”

Differential auditory and visual phase-locking are observed during audio-visual benefit and silent lip-reading for speech perception

“…A recent study using electrocorticography similarly demonstrated that medial occipital cortex exhibits reliable auditory envelope tracking in the absence of visual speech (Micheli et al, 2020). Other studies have suggested that visual cortex represents unheard auditory speech during silent lipreading by tracking its amplitude envelope (Hauswald et al, 2018) and higher-level linguistic feature representations (Nidiffer et al, 2021; Suess et al, 2022). Correspondingly, we also found evidence of visual cortex tracking the unheard auditory speech envelope in silent lipreading (Fig.…”