Distinct Computational Principles Govern Multisensory Integration in Primary Sensory and Association Cortices

Abstract

Human observers typically integrate sensory signals in a statistically optimal fashion into a coherent percept by weighting them in proportion to their reliabilities. An emerging debate in neuroscience is to which extent multisensory integration emerges already in primary sensory areas or is deferred to higher-order association areas. This fMRI study used multivariate pattern decoding to characterize the computational principles that define how auditory and visual signals are integrated into spatial representations across the cortical hierarchy. Our results reveal small multisensory influences that were limited to a spatial window of integration in primary sensory areas. By contrast, parietal cortices integrated signals weighted by their sensory reliabilities and task relevance in line with behavioral performance and principles of statistical optimality. Intriguingly, audiovisual integration in parietal cortices was attenuated for large spatial disparities when signals were unlikely to originate from a common source. Our results demonstrate that multisensory interactions in primary and association cortices are governed by distinct computational principles. In primary visual cortices, spatial disparity controlled the influence of non-visual signals on the formation of spatial representations, whereas in parietal cortices, it determined the influence of task-irrelevant signals. Critically, only parietal cortices integrated signals weighted by their bottom-up reliabilities and top-down task relevance into multisensory spatial priority maps to guide spatial orienting.

The Impact of Feedback on the Different Time Courses of Multisensory Temporal Recalibration

“…At the neural level, this is in line with studies of visual perceptual learning that observed changes in activity patterns in the anterior cingulate cortex to track changes in decision-making during visual perceptual learning [34, 37]. Furthermore, neural evidence suggests that prediction error signals during perceptual learning refine and strengthen neural connectivity between sensory neurons and those neurons required for the perceptual response and thus may support changes in higher-order regions [58]. Thus, in the absence of an informative reinforcement signal, rapid but transient changes in perceptual plasticity are likely due to changes in low-level sensory areas.…”

“…Thus, in the absence of an informative reinforcement signal, rapid but transient changes in perceptual plasticity are likely due to changes in low-level sensory areas. Future investigations will be necessary to determine if changes in the connectivity of higher-order cortical areas and low-level sensory processes underlie the observed changes in temporal recalibration and if these changes are durable or transient (see [58] for a helpful review in this regard).…”

Sounds facilitate visual motion discrimination via the enhancement of late occipital visual representations

“…However, recent studies suggest that there may be no generic answer to this question, as multisensory processing likely involves a distributed set of task- and function-specific regions (Bizley et al, 2016, Werner and Noppeney, 2010). In line with this hypothesis, two recent fMRI studies have illustrated how the computational nature of Audio-visual interactions changes from low-level sensory to high-level parietal cortices (Rohe and Noppeney, 2014, Rohe and Noppeney, 2016). …”

“…While there is an emerging consensus that the underlying neural correlates likely involve multiple stages of the sensory decision making pathways, it remains a challenge to uncover the dynamic processes that implement the multisensory benefit for an upcoming decision in the human brain (Bizley et al, 2016, Kayser and Shams, 2015, Rohe and Noppeney, 2014, Rohe and Noppeney, 2016). For example, many studies have shown that judgements about visual motion can be influenced by simultaneous sounds (Alais and Burr, 2004, Beer and Roder, 2004, Lewis and Noppeney, 2010, Schmiedchen et al, 2012) or vestibular information (Fetsch et al, 2010, Gu et al, 2008), even so when the multisensory stimulus is not directly task relevant (Gleiss and Kayser, 2014b, Kim et al, 2012, Sekuler et al, 1997).…”

“…While this could be taken to suggest that multisensory benefits for visual motion discrimination in the human brain are similarly arising from an enhancement of the encoding of visual motion in occipital regions, we still have a limited understanding of when and where the underlying neural processes operate. While fMRI studies support a central role of visual motion cortex in mediating multisensory benefits (Alink et al, 2008, Lewis and Noppeney, 2010, Scheef et al, 2009), studies on other tasks such as spatial localization have provided a more nuanced picture, one in which multiple occipital and parietal regions contribute distinctively to multisensory integration (Rohe and Noppeney, 2014, Rohe and Noppeney, 2016). For example, while studies using planar motion have implied the hMT complex (but see (Baumann and Greenlee, 2007)), a study on motion in depth has pointed to a role of area V3A (Ogawa and Macaluso, 2013) and regions within the IPS (Guipponi et al, 2013).…”