Causal inference regulates audiovisual spatial recalibration via its influence on audiovisual perception

Abstract

To obtain a coherent perception of the world, our senses need to be in alignment. When we encounter misaligned cues from two sensory modalities, the brain must infer which cue is faulty and recalibrate the corresponding sense. We examined whether and how the brain uses cue reliability to identify the miscalibrated sense by measuring the audiovisual ventriloquism aftereffect for stimuli of varying visual reliability. To adjust for modality-specific biases, visual stimulus locations were chosen based on perceived alignment with auditory stimulus locations for each participant. During an audiovisual recalibration phase, participants were presented with bimodal stimuli with a fixed perceptual spatial discrepancy; they localized one modality, cued after stimulus presentation. Unimodal auditory and visual localization was measured before and after the audiovisual recalibration phase. We compared participants’ behavior to the predictions of three models of recalibration: (a) Reliability-based: each modality is recalibrated based on its relative reliability—less reliable cues are recalibrated more; (b) Fixed-ratio: the degree of recalibration for each modality is fixed; (c) Causal-inference: recalibration is directly determined by the discrepancy between a cue and its estimate, which in turn depends on the reliability of both cues, and inference about how likely the two cues derive from a common source. Vision was hardly recalibrated by audition. Auditory recalibration by vision changed idiosyncratically as visual reliability decreased: the extent of auditory recalibration either decreased monotonically, peaked at medium visual reliability, or increased monotonically. The latter two patterns cannot be explained by either the reliability-based or fixed-ratio models. Only the causal-inference model of recalibration captures the idiosyncratic influences of cue reliability on recalibration. We conclude that cue reliability, causal inference, and modality-specific biases guide cross-modal recalibration indirectly by determining the perception of audiovisual stimuli.

Short-term monocular deprivation boosts neural responsiveness to audio-visual events for the undeprived eye

“…However, at the behavioral level, a tendency for aninteraction between illusion perception and sensory preferences was found (F(1,15)=4.01, p=0.06), indicating that Visual subjects tended to experience less illusion than Audio subjects, especially in the Undeprived eye (see section Audio-visual in the Behavioral Results). Interestingly, the perception of the multisensory input is known to partially depend on individual sensory predisposition (e.g., Giard and Perronet, 1999; Hong et al, 2021). Therefore, despite this was not the focus of the paper, we additionally explored whether the association between neural and behavioral changes might be affected by individual sensory preference.…”

“…This inference should be further verified with psychophysics experiments (see Rohe et al, 2019), designed to directly estimate auditory modality’s weight changes during audio-visual processing after MD. Interestingly, a recent study investigating cross-modal recalibration highlighted how individual variability in one sensory modality (visual reliability) differently affects recalibration of the other modality (audition) (Hong et al, 2021).…”

Audiovisual spatial recalibration but not integration is shaped by early sensory experience

“… Zaidel et al. (2011) suggested that crossmodal recalibration uses a fixed-ratio weighting of the crossmodal cues which is independent of current cue reliability (but see recent results from Hong et al., 2021 ). Although multisensory integration aims at improving precision and, thus, should take cue reliability into account, crossmodal recalibration might aim at achieving internal consistency (i.e., accuracy) of individual cues which does not depend on reliability ( Zaidel et al., 2011 ).…”

“…A study in children has recently found that crossmodal spatial recalibration does not arise before middle childhood (with immediate recalibration occurring before cumulative recalibration) and, thus, develops later than the ability to integrate auditory and visual spatial input for localization ( Rohlf et al., 2020 ). Moreover, it has been suggested that the weights assigned to each sensory cue during cumulative recalibration in adulthood do not depend on their precision but rather seem to be fixed ( Rohlf et al., 2021 ; Zaidel et al., 2011 ; but see Hong et al., 2021 ), whereas they are dependent on cue reliability in both immediate recalibration and multisensory integration ( Rohlf et al., 2021 ). Thus, the late developmental onset of cumulative crossmodal recalibration might be a consequence of the need to acquire the relative weighting of individual sensory cues through extensive experience which is consecutively stabilized by an elaboration of anatomical connectivity during development ( Linkenhoker et al., 2005 ).…”

Repeated exposure to either consistently spatiotemporally congruent or consistently incongruent audiovisual stimuli modulates the audiovisual common-cause prior

“…Körding and colleagues 17 formalized how an ideal Bayesian observer makes a perceptual judgment using causal inference. This Bayesian causal-inference model captures human behavior in a variety of multisensory tasks 8,[17][18][19][20][21][22][23][24][25][26][27][28] . According to this model, the observer derives the posterior probability of the two measurements coming from a common source.…”