The processing of audio-visual speech: empirical and neural bases

Abstract

In this selective review, I outline a number of ways in which seeing the talker affects auditory perception of speech, including, but not confined to, the McGurk effect. To date, studies suggest that all linguistic levels are susceptible to visual influence, and that two main modes of processing can be described: a complementary mode, whereby vision provides information more efficiently than hearing for some under-specified parts of the speech stream, and a correlated mode, whereby vision partially duplicates information about dynamic articulatory patterning.Cortical correlates of seen speech suggest that at the neurological as well as the perceptual level, auditory processing of speech is affected by vision, so that 'auditory speech regions' are activated by seen speech. The processing of natural speech, whether it is heard, seen or heard and seen, activates the perisylvian language regions (leftOright). It is highly probable that activation occurs in a specific order. First, superior temporal, then inferior parietal and finally inferior frontal regions (leftOright) are activated. There is some differentiation of the visual input stream to the core perisylvian language system, suggesting that complementary seen speech information makes special use of the visual ventral processing stream, while for correlated visual speech, the dorsal processing stream, which is sensitive to visual movement, may be relatively more involved.

Neural correlates of audiovisual speech processing in a second language

“…In the first step, L1 showed a subadditive effect in the bilateral inferior frontal gyrus (IFG, MNI coordinates; 54, 15, 27 and -54, 21, 27; corrected for multiple comparisons using the Monte Carlo simulations in AFNI). This region is commonly associated with multisensory processing, including AV speech processing (Lee & Noppeney, 2011;Calvert 2001;Campbell, 2008). However in the second step, this region did not survive either the max or the minimum criterion.…”

“…Most notably, a considerable body of evidence has associated the posterior part of the superior temporal sulcus (pSTS) with AV integration during language processing (for reviews, see Amedi et al, 2005;Beauchamp, 2005;Campbell, 2008). This cortical region responds to both visual and auditory speech stimuli and, more importantly, it often shows stronger responses when speech stimuli are simultaneously presented in the two sensory modalities (e.g., speech with co-occurring and correlated mouth movements, amongst others, usually meaningful stimuli).…”

“…This cortical region responds to both visual and auditory speech stimuli and, more importantly, it often shows stronger responses when speech stimuli are simultaneously presented in the two sensory modalities (e.g., speech with co-occurring and correlated mouth movements, amongst others, usually meaningful stimuli). The enhancement effect has been highlighted to be a key aspect that defines the responses of the STS and other regions of multisensory integration (Beauchamp, 2005;Calvert et al, 2000;Campbell, 2008).…”

“…Note, however, that from a logical (and empirical) point of view, the reverse is not necessarily true; that is, not all multisensory regions might be sensitive to stimulus congruency in one domain or more (Campbell, 2008).…”

Audio-visual speech perception in infants and toddlers with Down syndrome, fragile X syndrome, and Williams syndrome

“…Speech perception is often multimodal (Campbell, 1996(Campbell, , 2008 visual attention] and the suggestion that multimodal information processing is atypical in FXS). Additionally, auditory memory is worse than visual memory in DS (e.g., Marcell & Armstrong, 1982), which indicates that, in at least some tasks, individuals rely more on one modality than another.…”