Cortical tracking of unheard formant modulations derived from silently presented lip movements and its decline with age

Abstract

In a previous work we showed that the visual cortex tracks acoustic amplitude modulations accompanying lip movements during silently presented visual speech. Whether similar visuo-phonological transformation processes also exist for spectral modulations is unknown, which altogether could support the integration of auditory and visual cues for speech processing. Also, given increasing hearing difficulties in elderly individuals, we were interested in how these processes change as a function of age. Participants watched silent videos of a speaker and paid attention to the lip movements while being seated in the MEG. We found that the visual cortex not only tracks the unheard speech envelope, but also the unheard modulations of resonant frequencies (or formants) and the pitch (or fundamental frequency) linked to the lip movements, a process that is in general related to speech comprehension. Interestingly, only the processing of intelligible unheard formants decreases significantly with age in the visual and also in the cingulate cortex. This is not the case for the processing of the unheard speech envelope, the fundamental frequency or the purely visual information carried by lip movements. These results show that not only the global unheard speech envelope, but also unheard spectral fine-details are transformed from a mere visual to a phonological representation. Aging affects especially the ability to derive spectral dynamics at formant frequencies. Since listening in noisy environments should capitalize on the ability to track spectral fine-details, our results provide a novel focus on compensatory processes in such challenging situations.

Occlusion of lip movements impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

“…Hauswald et al (2018)). While Hauswald et al (2018) proposed a role of the visually conveyed envelope information for a visuo-phonological transformation, another study by our group further suggests that also the visually transported formant information go through such a transformation (Suess, Hauswald, et al, 2021), which is possibly even more relevant for visual speech processing than the transformation of the speech envelope (Plass et al, 2020). Finally, the reconstruction of voice pitch or fundamental frequency, used to segregate concurrent speech streams (Bregman, 1990), is also reduced through face masks, which might lead to difficulties disentangling the target speech stream and the distractor speech stream.…”

“…In addition to these effects in auditory processing regions, we have provided evidence for a direct visuo-phonological transformation when individuals only process visual information (i.e. silent video recordings of speakers), by showing that the acoustic speech envelope is tracked in visual cortical regions when individuals observe lip movements (Hauswald et al, 2018;Suess, Hauswald, et al, 2021). Furthermore, when again provided with only visual speech, the visual cortex also tracks spectral modulations in the range of the pitch, as well as in the second (F2) and third formant (F3), which reflect mainly sounds produced with the visible part of the mouth (Suess, Hauswald, et al, 2021).…”

“…Additionally, we investigated the averaged F2 and F3 (F2/3), as these two formants generated in the front cavity converge into 'focal points' after specific vowel-consonant combinations (Badin et al, 1990) and their frequency range has been shown to correlate strongly with lip movements (Chandrasekaran et al, 2009). Furthermore, this speech feature has been shown to be tracked by visual-only speech and is therefore prone to be affected through face masks (Suess, Hauswald, et al, 2021). Detailed results for F1, F2, F3 are depicted in Table S1 (see Supplementary Material).…”

“…silent video recordings of speakers), by showing that the acoustic speech envelope is tracked in visual cortical regions when individuals observe lip movements (Hauswald et al, 2018;Suess, Hauswald, et al, 2021). Furthermore, when again provided with only visual speech, the visual cortex also tracks spectral modulations in the range of the pitch, as well as in the second (F2) and third formant (F3), which reflect mainly sounds produced with the visible part of the mouth (Suess, Hauswald, et al, 2021). These results align well with previous findings by Chandrasekaran et al (2009), who indicate that the area of mouth opening correlates strongest with spectral components of speech in the range of 1 kHz -3 kHz, corresponding to the frequency range of F2 and F3.…”