Spatiotemporal dynamics of audiovisual speech processing

Abstract

The cortical processing of auditory-alone, visual-alone, and audiovisual speech information is temporally and spatially distributed, and functional magnetic resonance imaging (fMRI) cannot adequately resolve its temporal dynamics. In order to investigate a hypothesized spatio-temporal organization for audiovisual speech processing circuits, event-related potentials (ERPs) were recorded using electroencephalography (EEG). Stimuli were congruent audiovisual /bα/, incongruent auditory /bα/ synchronized with visual /gα/, auditory-only /bα/, and visual-only /bα/ and /gα/. Current density reconstructions (CDRs) of the ERP data were computed across the latency interval of 50-250 milliseconds. The CDRs demonstrated complex spatio-temporal activation patterns that differed across stimulus conditions. The hypothesized circuit that was investigated here comprised initial integration of audiovisual speech by the middle superior temporal sulcus (STS), followed by recruitment of the intraparietal sulcus (IPS), followed by activation of Broca's area (Miller and d'Esposito, 2005). The importance of spatio-temporally sensitive measures in evaluating processing pathways was demonstrated. Results showed, strikingly, early (< 100 msec) and simultaneous activations in areas of the supramarginal and angular gyrus (SMG/AG), the IPS, the inferior frontal gyrus, and the dorsolateral prefrontal cortex. Also, emergent left hemisphere SMG/AG activation, not predicted based on the unisensory stimulus conditions was observed at approximately 160 to 220 msec. The STS was neither the earliest nor most prominent activation site, although it is frequently considered the sine qua non of audiovisual speech integration. As discussed here, the relatively late activity of the SMG/AG solely under audiovisual conditions is a possible candidate audiovisual speech integration response.

The neural substrates for atypical planning and execution of word production in stuttering

“…This role is consistent with previous evidence that the AG is involved in the self-monitoring network for speech and plays an integrative role in motor control (Bernstein et al, 2008;Christoffels et al, 2007;Penhune et al, 1998;Shahin et al, 2009). Compared to non-stuttering speakers, stuttering speakers showed stronger positive connections from the left and right IFG, but weaker negative input from the left PMA.…”

Neural processing of emotion in multimodal settings

“…Visual dominance in emotion processing was also reported by Regenbogen et al (2013). Here, the presence of facial emotions enhanced functional connectivity between the FFA and areas of the angular gyrus associated with audiovisual speech integration (Bernstein et al, 2008). Neural systems thus seem to prioritize emotional over neutral facial information.…”

Dual Neural Routing of Visual Facilitation in Speech Processing

“…Regarding parietal and prefrontal input to auditory cortex, dynamic source reconstruction of responses to visual syllables showed that each of these regions responded shortly after stimulus presentation (yet later than motion-sensitive cortex) (supplemental Fig. 3, available at www.jneurosci.org as supplemental material) (Bernstein et al, 2008b). The STS was the latest of these three regions to respond, and it peaked simultaneously with the auditory cortex ϳ20 ms after the motion-sensitive occipital cortex.…”