Neural processing of asynchronous audiovisual speech perception

Abstract

The temporal synchrony of auditory and visual signals is known to affect the perception of an external event, yet it is unclear what neural mechanisms underlie the influence of temporal synchrony on perception. Using parametrically varied levels of stimulus asynchrony in combinations with BOLD fMRI, we identified two anatomically distinct subregions of multisensory superior temporal cortex (mSTC) that showed qualitatively distinct BOLD activation patterns. A synchrony-defined subregion of mSTC (synchronous > asynchronous) responded only when auditory and visual stimuli were synchronous, whereas a bimodal subregion of mSTC (auditory > baseline and visual > baseline) showed significant activation to all presentations, but showed monotonically increasing activation with increasing levels of asynchrony. The presence of two distinct activation patterns suggests that the two subregions of mSTC may rely on different neural mechanisms to integrate audiovisual sensory signals. An additional whole-brain analysis revealed a network of regions responding more with synchronous than asynchronous speech, including right mSTC, and bilateral superior colliculus, fusiform gyrus, lateral occipital cortex, and extrastriate visual cortex. The spatial location of individual mSTC ROIs was much more variable in the left than right hemisphere, suggesting that individual differences may contribute to the right lateralization of mSTC in a group SPM. These findings suggest that bilateral mSTC is composed of distinct multisensory subregions that integrate audiovisual speech signals through qualitatively different mechanisms, and may be differentially sensitive to stimulus properties including, but not limited to, temporal synchrony.

Top-down attention regulates the neural expression of audiovisual integration

“…Two main findings arose from this study. The first one was related to the superior temporal sulcus (STS), an area classically related to multisensory integration in and outside the speech domain (Beauchamp et al, 2004a;Calvert et al, 2000;Fairhall and Macaluso, 2009;Miller and D'Esposito, 2005;Nath and Beauchamp, 2012;Noesselt et al, 2012;Beauchamp et al, 2004aBeauchamp et al, , 2012Stevenson et al, 2010;Stevenson et al, 2011;Stevenson and James, 2009). This study showed a higher BOLD response in the STS when participants focused their visual spatial attention on the lips that were congruent with the auditory stream than when they focused their attention toward the location of the incongruent lips.…”

“…The STS has been often highlighted as one of the main loci for MSI, especially but not exclusively, regarding AV integration of speech (Beauchamp et al, 2004a(Beauchamp et al, , 2004bCalvert et al, 2000;Fairhall and Macaluso, 2009;Miller and D'Esposito, 2005;Nath and Beauchamp, 2012;Noesselt et al, 2007;Noesselt et al, 2012;Stevenson et al, 2010Stevenson et al, , 2011Stevenson and James, 2009). In particular, two different parts of the STS have been found to activate depending on the nature of the audiovisual manipulation.…”

“…For example, several high-level association brain areas have been previously highlighted as possible nodes of a MSI network (for reviews see, Beauchamp, 2005;Calvert, 2001;Campbell et al, 1996) One candidate area was the STS, a region that has been proved in many past studies to be responsive to multisensory stimulation in speech and other, meaningful, stimuli (Beauchamp et al, 2004a;Calvert et al, 2000;Fairhall and Macaluso, 2009;Miller and D'Esposito, 2005;Nath and Beauchamp, 2012;Noesselt et al, 2007;Noesselt et al, 2012;Stevenson et al, 2010;Stevenson et al, 2011;Stevenson and James, 2009). We hypothesized that we should see a higher BOLD response in the STS in the AV congruent condition as compared to the AV incongruent and control conditions.…”

Preference for Audiovisual Speech Congruency in Superior Temporal Cortex

“…It should be noted though that even during congruent speech auditory and visual signals are not necessarily temporally synchronous (Schwartz & Savariaux, 2014;Chandrasekaran, Trubanova, Stillittano, Caplier, & Ghazanfar, 2009), and speech signals are still perceived as congruent if the temporal asynchrony does not exceed a certain maximum ( Van Wassenhove, Grant, & Poeppel, 2007). The STC is generally susceptible to these temporal differences between congruent and incongruent speech (Stevenson, Altieri, Kim, Pisoni, & James, 2010). …”

Putative mechanisms mediating tolerance for audiovisual stimulus onset asynchrony

“…Possible brain regions involved in the reweighting mechanism include networks that quantify temporal synchrony and perceptual binding of AV percepts. These regions span the primary cortices as well as the STC and regions along the temporal-parietal-occipital junction (Beauchamp et al 2004a(Beauchamp et al , 2004bBushara et al 2001;Calvert et al 2000;Liebenthal et al 2005;Macaluso et al 2004;Powers et al 2009;Stevenson et al 2010Stevenson et al , 2011. The classical view posits that this occurs via a central integrator, the pSTS-G (Beauchamp et al 2004a(Beauchamp et al , 2004bCalvert and Campbell 2003;Keil et al 2011;Miller and Fig.…”