Reliability-Weighted Integration of Audiovisual Signals Can Be Modulated by Top-down Attention

Abstract

Behaviorally, it is well established that human observers integrate signals near-optimally weighted in proportion to their reliabilities as predicted by maximum likelihood estimation. Yet, despite abundant behavioral evidence, it is unclear how the human brain accomplishes this feat. In a spatial ventriloquist paradigm, participants were presented with auditory, visual, and audiovisual signals and reported the location of the auditory or the visual signal. Combining psychophysics, multivariate functional MRI (fMRI) decoding, and models of maximum likelihood estimation (MLE), we characterized the computational operations underlying audiovisual integration at distinct cortical levels. We estimated observers’ behavioral weights by fitting psychometric functions to participants’ localization responses. Likewise, we estimated the neural weights by fitting neurometric functions to spatial locations decoded from regional fMRI activation patterns. Our results demonstrate that low-level auditory and visual areas encode predominantly the spatial location of the signal component of a region’s preferred auditory (or visual) modality. By contrast, intraparietal sulcus forms spatial representations by integrating auditory and visual signals weighted by their reliabilities. Critically, the neural and behavioral weights and the variance of the spatial representations depended not only on the sensory reliabilities as predicted by the MLE model but also on participants’ modality-specific attention and report (i.e., visual vs. auditory). These results suggest that audiovisual integration is not exclusively determined by bottom-up sensory reliabilities. Instead, modality-specific attention and report can flexibly modulate how intraparietal sulcus integrates sensory signals into spatial representations to guide behavioral responses (e.g., localization and orienting).

Shared neural underpinnings of multisensory integration and trial-by-trial perceptual recalibration

“…Our data indeed support this conclusion, as parietal activity within the audio-visual trial was encoding the behaviorally combined information more than the acoustic information. Previous work has shown that parietal activity combines multisensory information flexibly depending on task-relevance and crossmodal disparity (30,34) and the focus on reporting the sound location in our task may have led to the attenuation of the combined visual information in the subsequent trial. Future work is required to better understand the influence of task-relevance on uni- and multisensory representations and how these are maintained over time.…”

“…These regions are in line with previous studies, which have pinpointed superior temporal regions, the insula and parietal-occipital areas as hubs for multisensory integration (11,26,27,47). In particular, a series of studies revealed that both temporal and parietal regions combine audio-visual information in a reliability- and task- dependent manner (30,31,34,48). However, while posterior parietal regions reflect the automatic fusion of multisensory information, more anterior parietal regions reflected an adaptive multisensory representation that follows predictions of Bayesian inference models.…”

“…Studies on spatial ventriloquist-like paradigms, for example, demonstrate contributions from auditory and parietal cortex (9,11,25–29). A series of recent studies demonstrates that posterior parietal regions automatically fuse multisensory information, while anterior parietal regions give way to a more flexible spatial representation that follows predictions from Bayesian causal inference (30–34). Given that parietal regions also contribute to the maintenance of sensory information within or between trials (35–40) this raises the possibility that parietal regions are in fact mediating both the combination of sensory information within a trial, and the influence of such an integrated representation on guiding subsequent adaptive behavior.…”

The neural dynamics of hierarchical Bayesian inference in multisensory perception

“…Collectively, fMRI and EEG research jointly suggest that computations involving unisensory estimates rely on lower-level regions at earlier latencies, while Bayesian Causal Inference estimates that take into account the world's causal structure arise later in higher-level cortical regions. Previous fMRI research implicated prefrontal cortices in the computations of the causal structure 28,39 , which may in turn inform the integration processes in parietal and temporal cortices 40 .…”

Altered functional connectivity during speech perception in congenital amusia

“…the "weighted connections" model of multisensory integration. In this model, the relative reliability of the AMUSIA AND LANGUAGE PROCESSING modalities involved with perception of a stimulus is related to differential connectivity strength (Beauchamp et al, 2010;Rohe and Noppeney, 2018). For example, when participants simultaneously view and feel touches to the hand and reliability of visual and tactile perception is manipulated experimentally via introduction of noise, connection strength (effective connectivity measured with functional MRI and structural equation modeling) between unimodal and multimodal sensory areas adjusts accordingly.…”