Seeing Is Believing: Neural Representations of Visual Stimuli in Human Auditory Cortex Correlate with Illusory Auditory Perceptions

Abstract

In interpersonal communication, the listener can often see as well as hear the speaker. Visual stimuli can subtly change a listener’s auditory perception, as in the McGurk illusion, in which perception of a phoneme’s auditory identity is changed by a concurrent video of a mouth articulating a different phoneme. Studies have yet to link visual influences on the neural representation of language with subjective language perception. Here we show that vision influences the electrophysiological representation of phonemes in human auditory cortex prior to the presentation of the auditory stimulus. We used the McGurk effect to dissociate the subjective perception of phonemes from the auditory stimuli. With this paradigm we demonstrate that neural representations in auditory cortex are more closely correlated with the visual stimuli of mouth articulation, which drive the illusory subjective auditory perception, than the actual auditory stimuli. Additionally, information about visual and auditory stimuli transfer in the caudal–rostral direction along the superior temporal gyrus during phoneme perception as would be expected of visual information flowing from the occipital cortex into the ventral auditory processing stream. These results show that visual stimuli influence the neural representation in auditory cortex early in sensory processing and may override the subjective auditory perceptions normally generated by auditory stimuli. These findings depict a marked influence of vision on the neural processing of audition in tertiary auditory cortex and suggest a mechanistic underpinning for the McGurk effect.

Spatio‐temporal distribution of brain activity associated with audio‐visually congruent and incongruent speech and the McGurk Effect

“…2008; Smith et al. 2013), may thus be activated by cortico‐cortical connections from the other affected areas. Whereas possible involvement of the mirror cell system (BA21) is evident from the very early period following consonant onset, possible motor involvement (gesture imitation, Mühlau et al.…”

“…Cortical visual inputs are only observable beginning at 100 msec, in secondary visual cortex (BA19) and Superior parietal cortex (BA7) in conjunction with increased activation of the Superior temporal cortex (BA22). Superior temporal cortex, most often associated with audio-visual speech processing (Wright et al 2003;Miller and d'Esposito 2005;Bernstein et al 2008;Smith et al 2013), may thus be activated by cortico-cortical connections from the other affected areas. Whereas possible involvement of the mirror cell system (BA21) is evident from the very early period following consonant onset, possible motor involvement (gesture imitation, Mühlau et al 2005) across all experimental conditions was only indicated during a limited period between 170 and 200 msec in the inferior parietal cortex (BA40).…”

Frequency-dependent representation of reinforcement-related information in the human medial and lateral prefrontal cortex

“…The FRN is most commonly observed on frontocentral EEG contacts, where its amplitude is highest (Miltner et al, 1997;Luu et al, 2003;Silvetti et al, 2014), but has also been observed on parietal contacts (Cohen and Ranganath, 2007;Pfabigan et al, 2011). EEG and MEG source localization has commonly identified the dorsomedial prefrontal cortex (dmPFC), especially the dorsal anterior cingulate cortex (dACC), as the source of the FRN (Miltner et al, 2003;Herrmann et al, 2004;Luu et al, 2004;Roger et al, 2010;Doñamayor et al, 2011;Walsh and Anderson, 2013).…”

The Effect of Amblyopia on Visual-Auditory Speech Perception

“…Speech perception is a multimodal phenomenon, integrating auditory input with the visual input of the speaker's oral and extraoral facial movement. [1][2][3] One of the most striking demonstrations of how vision influences the perception of sound is the McGurk effect. 4 In our demonstration, when presented with an audio track playing /pa/ and a separate video track of a person silently articulating /ka/, participants will most often claim to hear a unique fusion sound /ta/ (Figure 1; Video).…”