Cross-modal Binding and Activated Attentional Networks during Audio-visual Speech Integration: a Functional MRI Study

Abstract

We evaluated the neural substrates of cross-modal binding and divided attention during audio-visual speech integration using functional magnetic resonance imaging. The subjects (n = 17) were exposed to phonemically concordant or discordant auditory and visual speech stimuli. Three different matching tasks were performed: auditory-auditory (AA), visual-visual (VV) and auditory-visual (AV). Subjects were asked whether the prompted pair were congruent or not. We defined the neural substrates for the within-modal matching tasks by VV-AA and AA-VV. We defined the cross-modal area as the intersection of the loci defined by AV-AA and AV-VV. The auditory task activated the bilateral anterior superior temporal gyrus and superior temporal sulcus, the left planum temporale and left lingual gyrus. The visual task activated the bilateral middle and inferior frontal gyrus, right occipito-temporal junction, intraparietal sulcus and left cerebellum. The bilateral dorsal premotor cortex, posterior parietal cortex (including the bilateral superior parietal lobule and the left intraparietal sulcus) and right cerebellum showed more prominent activation during AV compared with AA and VV. Within these areas, the posterior parietal cortex showed more activation during concordant than discordant stimuli, and hence was related to cross-modal binding. Our results indicate a close relationship between cross-modal attentional control and cross-modal binding during speech reading.

Gender-Specific Effects of Prenatal and Adolescent Exposure to Tobacco Smoke on Auditory and Visual Attention

“…Plots of the percent signal change data revealed that, across exposure groups, activation of these regions was greater during the auditory conditions than during the visual conditions, consistent with earlier comparisons of auditory and visual attention demonstrating the importance of these regions for auditory attention (Saito et al, 2005;Shaywitz et al, 2001). Post hoc analyses indicated that during the auditory conditions, activation of right superior temporal gyrus was significantly greater in all exposure groups relative to subjects with neither prenatal nor adolescent exposure, whereas activation of left and right lingual gyrus was significantly greater in subjects with prenatal exposure to tobacco smoke alone than in subjects with neither prenatal nor adolescent exposure.…”

“…Consistent with the notion that in humans, neurocircuitry supporting auditory attention may be more vulnerable to the developmental toxicity of nicotine than neurocircuitry supporting visual attention, functional-imaging data revealed significant prenatal exposure by adolescent smoking by modality effects in regions that support auditory attention, including bilateral superior temporal gyrus, encompassing primary auditory cortex on the left (Saito et al, 2005;Shaywitz et al, 2001). During performance of the auditory attention tasks, activation of right superior temporal gyrus was greater in all exposure groups relative to subjects with neither prenatal nor adolescent exposure, whereas activation of bilateral lingual gyrus was greater in subjects with prenatal exposure to tobacco smoke alone relative to subjects with no exposure.…”

Corticocortical Connections Mediate Primary Visual Cortex Responses to Auditory Stimulation in the Blind

“…The auditory discrimination task we used led to significant activation in bilateral medial geniculate nucleus, primary auditory cortex, and primary visual cortex in both groups. The activation of primary visual cortex in response to auditory stimulation in the sighted might seem surprising at first sight, but note that previous studies have shown nonvisual responses in primary visual cortex of sighted participants (Maeder et al, 2001;Saito et al, 2005;Cate et al, 2009). More generally, both monkey and human studies have shown that traditional views of unimodal primary cortices have to be reconsidered, as primary cortices also respond to stimuli from other modalities, although to a lesser degree (for review, see Driver and Noesselt, 2008).…”

Touch, sound and vision in human superior temporal sulcus

“…The STS is also important for processing speech, one of the main auditory cues used by humans to communicate (Price, 2000), with a special role for the integration of auditory and visual language cues (Callan et al, 2004;Calvert et al, 2000;Macaluso et al, 2004;Miller and D'Esposito, 2005;Saito et al, 2005;Schroeder et al, 2008;Sekiyama et al, 2003;van Atteveldt et al, 2007). STSms prefers real auditory stimuli to scrambled auditory stimuli (Beauchamp et al, 2004b) consistent with its role in the representation of sensory stimuli with meaning for the individual.…”