Accumulation and decay of visual capture and the ventriloquism aftereffect caused by brief audio-visual disparities

Abstract

Visual capture and the ventriloquism aftereffect resolve spatial disparities of incongruent auditory-visual (AV) objects by shifting auditory spatial perception to align with vision. Here, we demonstrated the distinct temporal characteristics of visual capture and the ventriloquism aftereffect in response to brief AV disparities. In a set of experiments, subjects localized either the auditory component of AV targets (A within AV) or a second sound presented at varying delays (1-20s) after AV exposure (A2 after AV). AV targets were trains of brief presentations (1 or 20), covering a ±30° azimuthal range, and with ±8° (R or L) disparity. We found that the magnitude of visual capture generally reached its peak within a single AV pair and did not dissipate with time, while the ventriloquism aftereffect accumulated with repetitions of AV pairs and dissipated with time. Additionally, the magnitude of the auditory shift induced by each phenomenon was uncorrelated across listeners and visual capture was unaffected by subsequent auditory targets, indicating that visual capture and the ventriloquism aftereffect are separate mechanisms with distinct effects on auditory spatial perception. Our results indicate that visual capture is a ‘sample-and-hold’ process that binds related objects and stores the combined percept in memory, whereas the ventriloquism aftereffect is a ‘leaky integrator’ process that accumulates with experience and decays with time to compensate for cross-modal disparities.

Shared neural underpinnings of multisensory integration and trial-by-trial perceptual recalibration

“…In fact, most studies have relied on prolonged adaptation to multisensory discrepancies. Hence these studies investigated long-term recalibration, which seems to be mechanistically distinct from the trial-by-trial recalibration investigated here (7,18,19,27,28). The study most closely resembling the present one suggested that the ventriloquist after-effect is mediated by an interaction of auditory and parietal regions (18), a network centered view also supported by work on the McGurk after-effect (14,50).…”

“…Despite many behavioral studies demonstrating the robustness of multisensory trial-by-trial recalibration (6,7,9), little is known about the underlying neural substrate. Reasoning that recalibration relies on the persistence of information about previous stimuli, our study was guided by the quantification of where in the brain sensory information experienced during one trial can be recovered during the subsequent trial.…”

“…In fact, multisensory information can be exploited to calibrate subsequent perception in the absence of external feedback (5,6). In a ventriloquist paradigm, for example, the sight of the puppet and the actor’s voice are combined when localizing the speech source, and both cues influence the localization of subsequent unisensory acoustic cues, if probed experimentally (7–13). This trial-by-trial recalibration of perception by previous multisensory information has been demonstrated for spatial cues, temporal cues, and speech signals (14–17).…”

Spatial receptive field shift by preceding cross‐modal stimulation in the cat superior colliculus

“…; Bosen et al . ). The visual system is assumed to predominate within the process of spatial perception, with the auditory system being more influential in temporal perception (Morein‐Zamir et al .…”

“…Taken together, these results suggest how spatial calibration of the different senses occurs at the neuronal level. Vision is normally considered to be the most accurate and reliable amongst all the senses, and has been found capable of providing a reference for the calibration of auditory spatial perception (Zwiers et al 2003;Bosen et al 2017). The visual system is assumed to predominate within the process of spatial perception, with the auditory system being more influential in temporal perception (Morein-Zamir et al 2003;Recanzone, 2003;Spence & Squire, 2003;Eagleman, 2008).…”