The visual speech head start improves perception and reduces superior temporal cortex responses to auditory speech

Karas, Patrick J.; Magnotti, John F.; Metzger, Brian; Zhu, Lin; Smith, Kristen B; Yoshor, Daniel; Beauchamp, Michael S.

doi:10.7554/elife.48116

Cited by 41 publications

(59 citation statements)

References 62 publications

(77 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We suggest that this benefit is likely to be driven by an increased reliance on the visual articulations which help the listener to understand the noisy speech content by constraining phoneme identity (Karas et al, 2019). In line with this, we also show that the phonemes that most contribute to these results are those that have particularly informative visual articulations (Fig.…”

Section: Enhanced Multisensory Integration Effects At the Phonetic-lesupporting

confidence: 81%

“…One recent perspective (Peelle and Sommers, 2015) suggests that these stages could include an early stage, where visual speech provides temporal cues about the acoustic signal (correlated mode), and a later stage, where visual cues that convey place and manner of articulation could be integrated with acoustic information to constrain lexical selection (complementary mode). Such early-stage integration could be mediated by direct projections from visual cortex that dynamically affect the sensitivity of auditory cortex (Grant and Seitz, 2000;Okada et al, 2013;Tye-Murray et al, 2011;Calvert et al, 1997), whereas for later-stage integration, articulatory visual cues could be combined with acoustic information in supramodal regions such as the STS (Beauchamp et al, 2004;Kayser and Logothetis, 2009;Zhu and Beauchamp, 2017;Karas et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Neurophysiological indices of audiovisual speech integration are enhanced at the phonetic level for speech in noise

O’Sullivan

Crosse

Liberto

et al. 2020

Preprint

View full text Add to dashboard Cite

Seeing a speaker's face benefits speech comprehension, especially in challenging listening conditions. This perceptual benefit is thought to stem from the neural integration of visual and auditory speech at multiple stages of processing, whereby movement of a speaker's face provides temporal cues to auditory cortex, and articulatory information from the speaker's mouth can aid recognizing specific linguistic units (e.g., phonemes, syllables). However it remains unclear how the integration of these cues varies as a function of listening conditions. Here we sought to provide insight on these questions by examining EEG responses to natural audiovisual, audio, and visual speech in quiet and in noise.Specifically, we represented our speech stimuli in terms of their spectrograms and their phonetic features, and then Significance Statement During conversation, visual cues impact our perception of speech. Integration of auditory and visual speech is thought to occur at multiple stages of speech processing and vary flexibly depending on the listening conditions.Here we examine audiovisual integration at two stages of speech processing using the speech spectrogram and a phonetic representation, and test how audiovisual integration adapts to degraded listening conditions. We find significant integration at both of these stages regardless of listening conditions, and when the speech is noisy, we find enhanced integration at the phonetic stage of processing. These findings provide support for the multistage integration framework and demonstrate its flexibility in terms of a greater reliance on visual articulatory information in challenging listening conditions.

show abstract

Section: Enhanced Multisensory Integration Effects At the Phonetic-lesupporting

confidence: 81%

Section: Introductionmentioning

confidence: 99%

Neurophysiological indices of audiovisual speech integration are enhanced at the phonetic level for speech in noise

O’Sullivan

Crosse

Liberto

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…An important question for further research is determining the mechanisms by which visual speech modulates responses to auditory speech in pSTG/S. One possibility is that the early arrival of visual information influences the responses of populations of neurons in pSTG/S selective for different speech sounds, exciting populations that are compatible with the viewed mouth movements and inhibiting populations that are incompatible with it (6). From a practical perspective, normalizing the response pattern in the service of better comprehension could be accomplished through real-time fMRI neurofeedback (25) or through brain-computer interfaces (26).…”

Section: Discussionmentioning

confidence: 99%

“…In non-human primates, recordings from single neurons in pSTG/S respond to both auditory and visual social communication signals (3)(4)(5). In humans, small populations of neurons in pSTG/S recorded with intracranial electrodes respond to both auditory and visual speech (6,7). While the idea that pSTG/S integrates visual speech information with noisy auditory speech in the service of comprehension seems reasonable, it is supported by only limited empirical evidence.…”

Section: Introductionmentioning

confidence: 99%

Linking Activity in Human Superior Temporal Cortex to Perception of Noisy Audiovisual Speech

Rennig

Beauchamp

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Significance StatementEnabling social interactions, including the production and perception of speech, is a key function of the human brain. Speech perception is a complex computational problem that the brain solves using both visual information from the talker's facial movements and auditory information from the talker's voice. Visual speech information is particularly important under noisy listening conditions when auditory speech is difficult or impossible to understand alone Regions of the human cortex in posterior superior temporal lobe respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech. We show that the pattern of activity in cortex reflects the successful multisensory integration of auditory and visual speech information in the service of perception.

show abstract

“…While this schema was codified in responses to simple auditory beep and visual flash stimuli in anesthetized animals (Stein and Stanford, 2008) it has also been applied to human brain responses to auditory and visual speech recorded with BOLD fMRI, scalp encephalogram (EEG) and intracranial EEG (iEEG) (Besle et al, 2004;van Wassenhove et al, 2005;Besle et al, 2008;Rhone et al, 2016;Karas et al, 2019). In these studies, responses to unisensory and multisensory speech were compared under the assumption that different sensory responses were the main driver of neural activity.…”

Section: Introductionmentioning

confidence: 99%

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

Metzger

Magnotti

Wang

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Acknowledgments:This research was supported by NIH R01NS065395 and R25NS070694. AbstractExperimentalists studying multisensory integration compare neural responses to multisensory stimuli with responses to the component modalities presented in isolation. This procedure is problematic for multisensory speech perception since audiovisual speech and auditory-only speech are easily intelligible but visual-only speech is not. To overcome this confound, we developed intracranial encephalography (iEEG) deconvolution. Individual stimuli always contained both auditory and visual speech but jittering the onset asynchrony between modalities allowed for the time course of the unisensory responses and the interaction between them to be independently estimated. We applied this procedure to electrodes implanted in human epilepsy patients (both male and female) over the posterior superior temporal gyrus (pSTG), a brain area known to be important for speech perception. iEEG deconvolution revealed sustained, positive responses to visual-only speech and larger, phasic responses to auditory-only speech. Confirming results from scalp EEG, responses to audiovisual speech were weaker than responses to auditoryonly speech, demonstrating a subadditive multisensory neural computation. Leveraging the spatial resolution of iEEG, we extended these results to show that subadditivity is most pronounced in more posterior aspects of the pSTG. Across electrodes, subadditivity correlated with visual responsiveness, supporting a model in visual speech enhances the efficiency of auditory speech processing in pSTG. The ability to separate neural processes may make iEEG deconvolution useful for studying a variety of complex cognitive and perceptual tasks. Significance statementUnderstanding speech is one of the most important human abilities. Speech perception uses information from both the auditory and visual modalities. It has been difficult to study neural responses to visual speech because visual-only speech is difficult or impossible to comprehend, unlike auditory-only and audiovisual speech. We used intracranial encephalography (iEEG) deconvolution to overcome this obstacle. We found that visual speech evokes a positive response in the human posterior superior temporal gyrus, enhancing the efficiency of auditory speech processing.

show abstract

The visual speech head start improves perception and reduces superior temporal cortex responses to auditory speech

Cited by 41 publications

References 62 publications

Neurophysiological indices of audiovisual speech integration are enhanced at the phonetic level for speech in noise

Neurophysiological indices of audiovisual speech integration are enhanced at the phonetic level for speech in noise

Linking Activity in Human Superior Temporal Cortex to Perception of Noisy Audiovisual Speech

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

Contact Info

Product

Resources

About