Reading Fluent Speech from Talking Faces: Typical Brain Networks and Individual Differences

Abstract

Listeners are able to extract important linguistic information by viewing the talker's face -a process known as 'speechreading'. Previous studies of speechreading present small closed sets of simple words and their results indicate that visual speech processing engages a wide network of brain regions in the temporal, frontal, and parietal lobes that are likely to underlie multiple stages of the receptive language system. The present study further explored this network in a large group of subjects by presenting naturally-spoken sentences which tap the richer complexities of visual speech processing. Four different baselines (blank screen, static face, non-linguistic facial gurning, and auditory speech) enabled us to determine the hierarchy of neural processing involved in speechreading and to test the claim that visual input reliably accesses sound-based representations in the auditory cortex.In contrast to passively viewing a blank screen, the static-face condition evoked activation bilaterally across the border of the fusiform gyrus and cerebellum, and in the medial superior frontal gyrus and left precentral gyrus (P < 0.05, whole brain corrected). With the static face as baseline, the gurning face evoked bilateral activation in the motion-sensitive region of the occipital cortex, whereas visual speech additionally engaged middle temporal 1 gyrus, inferior and middle frontal gyri, and the inferior parietal lobe, particularly in the left hemisphere. These latter regions are implicated in lexical stages of spoken language processing. While auditory speech generated extensive bilateral activation across both superior and middle temporal gyri, the group-averaged pattern of speechreading activation failed to include any auditory regions along the superior temporal gyrus, suggesting that fluent visual speech does not always involve sound-based coding of the visual input.An important finding from the individual subject analyses was that activation in the superior temporal gyrus did reach significance (P < 0.001, small-volume corrected) for a subset of the group. Moreover, the extent of the left-sided superior temporal gyrus activity was strongly correlated with speechreading performance. Skilled speechreading was also associated with activations and deactivations in other brain regions suggesting that individual differences reflect the efficiency of a circuit linking sensory, perceptual, memory, cognitive, and linguistic processes rather than the operation of a single component process.

A Retrospective Multicenter Study Comparing Speech Perception Outcomes for Bilateral Implantation and Bimodal Rehabilitation

“…The analyses did not show any effect of side on speech performance, even at later ages when the right ear advantage for speech may increase (Martin & Jerger 2005). From the literature, the left hemispheric dominance for speech does not seem to be modified by deafness: (1) this hemispheric specialization is preserved in sign language processing MacSweeney et al 2008); (2) postlingual deaf subjects, even after years of profound deafness, preserve the left dominance for phonology processing (Lazard et al 2010(Lazard et al , 2012c; and (3) lipreading also shares left auditory cortical areas (Calvert & Campbell 2003;Hall et al 2005;Lazard et al 2014).…”

Adaptive benefit of cross-modal plasticity following cochlear implantation in deaf adults

“…Rather, our results provide evidence that increased cross-modal activation of auditory brain regions by visual speech may offer a facilitative link between the two modalities that promotes auditory recovery after cochlear implantation. Cross-modal activation of superior temporal cortex by visual speech may reflect processes such as inner speech and auditory imagery due to the inherent correspondence that exists between auditory and visual speech representations (20). In this way, an increase in STC activation to visual speech may reflect a stronger correspondence or synergy between the modalities that may facilitate auditory recovery.…”

“…The Institute of Hearing Research (IHR) Number Sentences (20) were presented as speech stimuli during the acquisition of fNIRS measurements. The corpus comprised digital audiovisual recordings of 90 sentences, each spoken by both a male and female talker.…”

Cued Speech for Enhancing Speech Perception and First Language Development of Children With Cochlear Implants

“…All of these areas are implicated in phonological processing, speech and mouth motor planning, and speech execution. Interestingly, activation in the portions of the superior temporal cortex appears to be modulated by speechreading skill in hearing participants (Hall, Fussell, & Summerfield, 2005). fMRI studies of natural audiovisual speech (MacSweeney et al, 2002;Calvert et al, 1997) also show consistent and extensive activation of the superior temporal gyrus in hearing people.…”