Quantifying temporal ventriloquism in audiovisual synchrony perception

Abstract

The integration of visual and auditory inputs in the human brain works properly only if the components are perceived in close temporal proximity. In the present study, we quantified cross-modal interactions in the human brain for audiovisual stimuli with temporal asynchronies, using a paradigm from rhythm perception. In this method, participants had to align the temporal position of a target in a rhythmic sequence of four markers. In the first experiment, target and markers consisted of a visual flash or an auditory noise burst, and all four combinations of target and marker modalities were tested. In the same-modality conditions, no temporal biases and a high precision of the adjusted temporal position of the target were observed. In the different-modality conditions, we found a systematic temporal bias of 25-30 ms. In the second part of the first and in a second experiment, we tested conditions in which audiovisual markers with different stimulus onset asynchronies (SOAs) between the two components and a visual target were used to quantify temporal ventriloquism. The adjusted target positions varied by up to about 50 ms and depended in a systematic way on the SOA and its proximity to the point of subjective synchrony. These data allowed testing different quantitative models. The most satisfying model, based on work by Maij, Brenner, and Smeets (Journal of Neurophysiology 102, 490-495, 2009), linked temporal ventriloquism and the percept of synchrony and was capable of adequately describing the results from the present study, as well as those of some earlier experiments.

Hearing flashes and seeing beeps: Timing audiovisual events

“…The perception of the visual component resulting from the AV interaction could easily be forward but only little backward in time by asynchronous beeps. This auditory capture asymmetry is consistent with an early study [ 5 ] but not with [ 26 ]. In the latter, the attraction of visual flashes by auditory beeps was stronger when beeps were leading than lagging.…”

“…When presented in proximity, the perceived timing of visual events shifts towards the real timing of auditory events while the reverse is negligible. Previous studies provided quantitative estimations of the temporal attractions using either bimodal probes [ 24 , 25 ] or unimodal visual probes [ 26 ]. Since in an unbiased situation auditory beeps should be played about 40ms after visual flashes to produce perceptual synchrony, using physical synchrony as a reference to measure time shifts [ 24 , 25 ] might not be suitable.…”

“…Since in an unbiased situation auditory beeps should be played about 40ms after visual flashes to produce perceptual synchrony, using physical synchrony as a reference to measure time shifts [ 24 , 25 ] might not be suitable. Only one study tried to quantify auditory capture across a large range of auditory-visual offsets [ 26 ], but the use of two separate groups of participants to measure interleaved sets of SOA points led to a surprising serrated pattern of results. Finally, measuring the converse effect–the visual capture of auditory events–was lacking in this study [ 26 ], thereby assuming a total auditory dominance in the sensory interaction.…”

“…In a different context, the temporal attraction of a flash by a click was only slightly stronger than the reverse (60ms against 42ms) in the SOA range of –100ms to +100ms [ 6 ]. Furthermore, the quantitative estimations of time shifts with a bisection task confirmed that audition dominates in the perceived timing of AV stimuli [ 24 – 26 ]. One question that this dominance raises is whether the cue combination in the temporal domain follows the same rules as in the spatial domain, that is optimal integration in the Bayesian framework according to which the weighting of each sensory modality is proportional to its reliability [ 32 ].…”

“…Top-up trials with the adapted SOA before each test proposed in [ 21 ] is a good solution to cancel these interactions, but this study suffers from the TOJ tasks limitations for timing described earlier. Finally, a rhythmic task where subjects were presented five events and had to adjust the temporal position of the middle one to align it with the tempo defined by the other four was used in [ 26 ]. Although this paradigm is a clear improvement of bisection tasks, this study suffers from several methodological flaws, starting with the use of an adjustment task which is far less reliable than forced-choices.…”