Cortical Hierarchies Perform Bayesian Causal Inference in Multisensory Perception

Abstract

To form a veridical percept of the environment, the brain needs to integrate sensory signals from a common source but segregate those from independent sources. Thus, perception inherently relies on solving the “causal inference problem.” Behaviorally, humans solve this problem optimally as predicted by Bayesian Causal Inference; yet, the underlying neural mechanisms are unexplored. Combining psychophysics, Bayesian modeling, functional magnetic resonance imaging (fMRI), and multivariate decoding in an audiovisual spatial localization task, we demonstrate that Bayesian Causal Inference is performed by a hierarchy of multisensory processes in the human brain. At the bottom of the hierarchy, in auditory and visual areas, location is represented on the basis that the two signals are generated by independent sources (= segregation). At the next stage, in posterior intraparietal sulcus, location is estimated under the assumption that the two signals are from a common source (= forced fusion). Only at the top of the hierarchy, in anterior intraparietal sulcus, the uncertainty about the causal structure of the world is taken into account and sensory signals are combined as predicted by Bayesian Causal Inference. Characterizing the computational operations of signal interactions reveals the hierarchical nature of multisensory perception in human neocortex. It unravels how the brain accomplishes Bayesian Causal Inference, a statistical computation fundamental for perception and cognition. Our results demonstrate how the brain combines information in the face of uncertainty about the underlying causal structure of the world.

The Curious Incident of Attention in Multisensory Integration: Bottom-up vs. Top-down

“…The latter will be determined by the type of sensory feature representation, as well as by prior knowledge and goals. The development of new theoretical and mathematical approaches (e.g., Bayesian causal inference, Rohe and Noppeney, 2015a) will help us with the interpretation of these neuronal effects. In addition, I believe that together with sophisticated and well-controlled experiments, it is important to look also into more naturalistic and life-like multisensory conditions.…”

Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex

“…Multi-voxel pattern analysis (MVPA) represents an opposite way of modeling, trying to predict stimulus categories using an entire hemodynamic activation pattern, without being restricted to an assumption of certain predefined response function or stimulus model (Norman et al 2006, Pereira et al 2008, Mur et al 2009). By enabling classification of complex stimulus-specific activation patterns even in the absence of regional amplitude changes, MVPA provides a powerful new approach to investigate the mechanisms of audiovisual integration (Pooresmaeli et al 2014, Gentile et al 2015, Li et al 2015, Rohe & Noppeney 2015. For instance, Li et al (2015) recently found distributed content-specific (male vs. female, crying vs. laughing) supratemporal activations during audiovisual perception of faces and voices during selective attention to particular features.…”

Disentangling signal and noise in autism spectrum disorder

“…If not, the system will attempt to reduce this mismatch, or prediction error, by adjusting its prediction about the state of the environment and adapting its generative model for the current context accordingly. Within this scheme, these models are hierarchically structured (Rohe & Noppeney, 2015;Wacongne et al, 2011), where higher levels are capable of capturing patterns in sensory inputs that have larger spatial or temporal spans.…”