Reading Speech from Still and Moving Faces: The Neural Substrates of Visible Speech

Abstract

Speech is perceived both by ear and by eye. Unlike heard speech, some seen speech gestures can be captured in stilled image sequences. Previous studies have shown that in hearing people, natural time-varying silent seen speech can access the auditory cortex (left superior temporal regions). Using functional magnetic resonance imaging (fMRI), the present study explored the extent to which this circuitry was activated when seen speech was deprived of its time-varying characteristics. In the scanner, hearing participants were instructed to look for a prespecified visible speech target sequence ("voo" or "ahv") among other monosyllables. In one condition, the image sequence comprised a series of stilled key frames showing apical gestures (e.g., separate frames for "v" and "oo" [from the target] or "ee" and "m" [i.e., from nontarget syllables]). In the other condition, natural speech movement of the same overall segment duration was seen. In contrast to a baseline condition in which the letter "V" was superimposed on a resting face, stilled speech face images generated activation in posterior cortical regions associated with the perception of biological movement, despite the lack of apparent movement in the speech image sequence. Activation was also detected in traditional speech-processing regions including the left inferior frontal (Broca's) area, left superior temporal sulcus (STS), and left supramarginal gyrus (the dorsal aspect of Wernicke's area). Stilled speech sequences also generated activation in the ventral premotor cortex and anterior inferior parietal sulcus bilaterally. Moving faces generated significantly greater cortical activation than stilled face sequences, and in similar regions. However, a number of differences between stilled and moving speech were also observed. In the visual cortex, stilled faces generated relatively more activation in primary visual regions (V1/V2), while visual movement areas (V5/MT+) were activated to a greater extent by moving faces. Cortical regions activated more by naturally moving speaking faces included the auditory cortex (Brodmann's Areas 41/42; lateral parts of Heschl's gyrus) and the left STS and inferior frontal gyrus. Seen speech with normal time-varying characteristics appears to have preferential access to "purely" auditory processing regions specialized for language, possibly via acquired dynamic audiovisual integration mechanisms in STS. When seen speech lacks natural time-varying characteristics, access to speech-processing systems in the left temporal lobe may be achieved predominantly via action-based speech representations, realized in the ventral premotor cortex.

Neural correlates of human action observation in hearing and deaf subjects

“…The activation of mouth motor regions during the perception of speech is well-attested (Wilson et al, 2004;Calvert & Campbell, 2003). The present results suggest homologous motor activations in the deaf, who primary language is articulated with the hands and fingers.…”

Spatio‐temporal distribution of brain activity associated with audio‐visually congruent and incongruent speech and the McGurk Effect

“…2005). In addition to the involvement of primary (Heschl's gyrus) and secondary auditory cortices (BA41,42) in auditory processing, they have also been mentioned in relation to audio‐visual speech perception (Calvert and Campbell 2003; Pekkola et al. 2005) and integration (Besle et al.…”

A Comparison of the Development of Audiovisual Integration in Children with Autism Spectrum Disorders and Typically Developing Children

“…Mirror neurone theory suggests that particular cells in the human superior temporal sulcus (STS) labelled 'mirror neurones', which are activated during passive observation of another person performing an action, do not function properly in autism, and that these cells are also involved in audiovisual integration (Williams et al, 2004). Impairment in mirror neurones in the STS of individuals with ASD might also explain deficits in speech-reading, as extensive activation of the STS during speech-reading tasks has consistently been shown in neuroimaging studies (Calvert & Campbell, 2003). There are reasons to suppose that mirror neurone systems may continue to develop well into adolescence (see Kilner & Blakemore, 2007), which would be consistent with the results of the current study.…”