Spatiotemporal models for multisensory integration

Abstract

Multisensory integration is a process of redundancy exploitation, in which our brains combine information across the senses to obtain more reliable perceptual estimates. While the high-level computational principles of multisensory integration are well understood, little is knowns as to how the low-level properties of the signals ultimately determine the integrated percept. This study demonstrates that a bottom-up approach, based on luminance- and sound-level analyses, is sufficient to jointly explain the spatiotemporal determinants of audiovisual integration and crossmodal attention. When implemented using an architecture analogous to the motion detectors found in the insect brain, such low-level analyses can broadly reproduce human behaviour–as tested in a large-scale simulation of 42 classic experiments on the spatial, temporal and attentional aspects of multisensory integration.

Multisensory integration operates on correlated input from unimodal transients channels

“…Indeed, the new MCD can near perfectly predict human performance in a causality judgment task with stimuli with a high temporal rate (up to 18Hz) using the same set of parameters optimized for stimuli with a much lower rate (5Hz), with a Pearson correlation coefficient of 0.99. Interestingly, recent results (Parise 2024) demonstrate that a population of spatially tuned MCD units can also account for the integration of ecological audiovisual stimuli over time and space, thereby replicating phenomena such as the McGurk Illusion, the Ventriloquist Illusion and even attentional orienting.…”