Multiple time scales of the ventriloquism aftereffect

Abstract

The ventriloquism aftereffect (VAE) refers to a shift in auditory spatial perception following exposure to a spatial disparity between auditory and visual stimuli. The VAE has been previously measured on two distinct time scales. Hundreds or thousands of exposures to a an audio-visual spatial disparity produces enduring VAE that persists after exposure ceases. Exposure to a single audio-visual spatial disparity produces immediate VAE that decays over seconds. To determine if these phenomena are two extremes of a continuum or represent distinct processes, we conducted an experiment with normal hearing listeners that measured VAE in response to a repeated, constant audio-visual disparity sequence, both immediately after exposure to each audio-visual disparity and after the end of the sequence. In each experimental session, subjects were exposed to sequences of auditory and visual targets that were constantly offset by +8° or −8° in azimuth from one another, then localized auditory targets presented in isolation following each sequence. Eye position was controlled throughout the experiment, to avoid the effects of gaze on auditory localization. In contrast to other studies that did not control eye position, we found both a large shift in auditory perception that decayed rapidly after each AV disparity exposure, along with a gradual shift in auditory perception that grew over time and persisted after exposure to the AV disparity ceased. We modeled the temporal and spatial properties of the measured auditory shifts using grey box nonlinear system identification, and found that two models could explain the data equally well. In the power model, the temporal decay of the ventriloquism aftereffect was modeled with a power law relationship. This causes an initial rapid drop in auditory shift, followed by a long tail which accumulates with repeated exposure to audio-visual disparity. In the double exponential model, two separate processes were required to explain the data, one which accumulated and decayed exponentially and the other which slowly integrated over time. Both models fit the data best when the spatial spread of the ventriloquism aftereffect was limited to a window around the location of the audio-visual disparity. We directly compare the predictions made by each model, and suggest additional measurements that could help distinguish which model best describes the mechanisms underlying the VAE.

Audio-visual interactions in egocentric distance perception: Ventriloquism effect and aftereffect

“…In terms of the temporal profile, the large immediate VAE and smaller persistent VAE observed here are generally consistent with the observations of multiple distinct cross-modal recalibration mechanisms reported in the horizontal VAE studies (Bosen, Fleming, Allen, O'Neill, & Paige, 2018;Watson, Akeroyd, Roach, & Webb, 2019). However, a signle adaptation mechanism cannot be ruled out for the current results given that the current study was not specifically designed to examine this question.…”

Distinct mechanisms govern recalibration to audio-visual discrepancies in remote and recent history

“…To further interrogate the mechanisms underlying the VAE, we fit the data using both exponential (leaky integrator) and power function models 24–26 . A leaky integrator predicts an exponential decay of the response characterised by two parameters: a trial-constant (τ) which determines the rate of change, with larger values giving a slower change, and a gain parameter which determines the overall response amplitude.…”

“…We also tested the ability of a power function model to predict the VAE across conditions, as such models can approximate the summation of multiple correlated exponential models 26–28 , and a recent study of the VAE suggests approximately similar performance between a multiple-exponential and a power model 24 . This model struggled to fully capture the pattern of responses across our conditions.…”

“…However, these behavioural effects pertain to rapid adaptation to the very recent past, hence it remains unclear whether distinct or unitary recalibration mechanisms exist over longer timescales. A recent study by Bosen and colleagues showed the growth and decay of the VAE could be predicted either by a multiple-exponential model or a power model 24 – yet it still remains unclear how these effects vary across different timescales, and whether opposing effects can be simultaneously maintained.…”

The neurophysiological basis of short- and long-term ventriloquism aftereffects

“…Despite the importance of such adaptive multisensory processes in everyday life, their neural underpinnings remain unclear. Our environment changes on multiple timescales and not surprisingly, perceptual recalibration emerges also on distinct scales (Bosen et al 2017(Bosen et al , 2018Röder 2015, 2019;Rohlf et al 2020;Van der Burg, Alais, and Cass 2015). During the ventriloquism aftereffect (Bruns and Röder 2015;Canon 1970;Radeau and Bertelson 1974;Recanzone 1998;Wozny and Shams 2011) the exposure to displaced acoustic and visual stimuli reliably biases the perceived location of subsequent unisensory sounds (Frissen, Vroomen, and de Gelder 2012;Mendonça et al 2015;Watson et al 2019;Woods and Recanzone 2004).…”