Abstract Coding of Audiovisual Speech: Beyond Sensory Representation

Abstract

Is there a neural representation of speech that transcends its sensory properties? Using fMRI, we investigated whether there are brain areas where neural activity during observation of sublexical audiovisual input corresponds to a listener's speech percept (what is "heard") independent of the sensory properties of the input. A target audiovisual stimulus was preceded by stimuli that (1) shared the target's auditory features (auditory overlap), (2) shared the target's visual features (visual overlap), or (3) shared neither the target's auditory or visual features but were perceived as the target (perceptual overlap). In two left-hemisphere regions (pars opercularis, planum polare), the target invoked less activity when it was preceded by the perceptually overlapping stimulus than when preceded by stimuli that shared one of its sensory components. This pattern of neural facilitation indicates that these regions code sublexical speech at an abstract level corresponding to that of the speech percept.

The Representation of Tool Use in Humans and Monkeys: Common and Uniquely Human Features

“…More generally, left aSMG site is located rostrally compared with the SMG regions involved in audio-visual phonological processing (Calvert and Campbell, 2003;Miller and D'Esposito, 2005;Hasson et al, 2007;. Similarly, action verbs compared with abstract verbs activate left area SII in the parietal operculum, rostral and ventral to the present aSMG site (Rüschemeyer et al, 2007).…”

Physical and Perceptual Factors Shape the Neural Mechanisms That Integrate Audiovisual Signals in Speech Comprehension

“…Hence the AV integration profile (i.e., whether it is suppressive or additive) in this left fronto-temporal circuitry is determined by the spectrotemporal structure of the signals and prior cross-modal training that enables articulatory and higher-order linguistic (e.g., phonological) representations. Recent studies of the McGurk effect have similarly shown that the left inferior frontal cortex codes participants' integrated AV McGurk percept ("da") rather than its visual ("ga") or auditory ("ba") constituents (Hasson et al, 2007;Skipper et al, 2007). Collectively, these results may suggest that the fronto-temporal circuitry integrates auditory and visual inputs into a speech percept guided by prior articulatory-gestural representations.…”

PET‐based investigation of cerebral activation following intranasal trigeminal stimulation

“…In addition, the parietal operculum has been shown to be involved in the orientation toward painful stimuli and in stimulus categorization and representation [Christmann et al, 2007;Dowman, 2007;Ohara et al, 2006]. With regard to the planum polare more research is needed on its possible role in the processing of intranasal trigeminal stimuli-previous studies mostly attributed activation in this area to the higher level processing of auditory information [e.g., Hasson et al, 2007].…”