Can you hear me yet? An intracranial investigation of speech and non-speech audiovisual interactions in human cortex

Abstract

In everyday conversation, viewing a talker's face can provide information about the timing and content of an upcoming speech signal, resulting in improved intelligibility. Using electrocorticography, we tested whether human auditory cortex in Heschl's gyrus (HG) and on superior temporal gyrus (STG) and motor cortex on precentral gyrus (PreC) were responsive to visual/gestural information prior to the onset of sound and whether early stages of auditory processing were sensitive to the visual content (speech syllable versus non-speech motion). Event-related band power (ERBP) in the high gamma band was content-specific prior to acoustic onset on STG and PreC, and ERBP in the beta band differed in all three areas. Following sound onset, we found with no evidence for content-specificity in HG, evidence for visual specificity in PreC, and specificity for both modalities in STG. These results support models of audio-visual processing in which sensory information is integrated in non-primary cortical areas.

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

“…In general, HG band responses were enhanced compared to baseline and LF responses were decreased compared to baseline and the dynamic visual speech information increased the HG response and reduced the LF response (enhanced decrease). Enhanced HG band responses and decreased LF responses compared to baseline have been previously reported to audiovisual speech with ECoG (Rhone et al, 2016;Schepers et al, 2015;Uno et al, 2015) and seem to be a general response profile in population electrophysiological responses to sensory stimulation (e.g., Scheeringa, Koopmans, van Mourik, Jensen, & Norris, 2016;Siegel, Donner, Oostenveld, Fries, & Engel, 2008). Magnitude changes in neural brain responses reported here, particularly in the broad HG range, which has been linked to neuronal spiking activity (Ray & Maunsell, 2011), might underlie BOLD response changes seen in previous fMRI studies (Mukamel et al, 2005).…”

“…Most studies on audiovisual speech perception so far compared average neural response magnitude and latency differences between unimodal and multimodal experimental conditions (Besle et al., ; Rhone et al., ; Schepers, Yoshor, & Beauchamp, ; van Wassenhove, Grant, & Poeppel, ) but did not focus on the representation of dynamic speech characteristics of the different modalities (e.g., using stimuli with continuous sentence analysis or using different perceptual modalities such as audio and video streams). The effects of visual speech information on auditory speech tracking have been investigated with EEG in auditory noise‐free (Crosse, Butler, & Lalor, ) and auditory noisy (Crosse, Di Liberto, & Lalor, ) conditions.…”

“…The posterior superior temporal gyrus and superior temporal sulcus (pSTG/pSTS) are likely candidates for auditory and visual speech tracking. These areas show multisensory responses to auditory and visual stimuli in fMRI (Beauchamp, Lee, Argall, & Martin, 2004;Lee & Noppeney, 2011;Noesselt et al, 2007;Stevenson & James, 2009), EEG/MEG (Arnal, Wyart, & Giraud, 2011;Schepers, Schneider, Hipp, Engel, & Senkowski, 2013) and with intracranial ECoG recordings (Ozker, Schepers, Magnotti, Yoshor, & Beauchamp, 2017;Rhone et al, 2016). However, the current understanding of audiovisual integration in the context of speech processing is rather incomplete.…”

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

“…A critical brain area for multisensory speech perception is the posterior superior temporal gyrus and sulcus (pSTG), the location of human auditory association cortex (Moerel et al, 2014;Leaver and Rauschecker, 2016). The belt and parabelt areas in pSTG are selective for both the complex acoustic-phonetic features that comprise auditory speech (Belin et al, 2000;Formisano et al, 2008;Mesgarani et al, 2014) and the mouth movements that comprise visual speech (Beauchamp et al, 2004;Bernstein et al, 2011;Rhone et al, 2016;Ozker et al, 2017;Zhu and Beauchamp, 2017;Ozker et al, 2018b;Rennig and Beauchamp, 2018;Beauchamp, 2019).…”

Linking Activity in Human Superior Temporal Cortex to Perception of Noisy Audiovisual Speech

“…In non-human primates, recordings from single neurons in pSTG/S respond to both auditory and visual social communication signals (3)(4)(5). In humans, small populations of neurons in pSTG/S recorded with intracranial electrodes respond to both auditory and visual speech (6,7). While the idea that pSTG/S integrates visual speech information with noisy auditory speech in the service of comprehension seems reasonable, it is supported by only limited empirical evidence.…”