Catching Audiovisual Interactions With a First-Person Fisherman Video Game

Abstract

The human brain is excellent at integrating information from different sources across multiple sensory modalities. To examine one particularly important form of multisensory interaction, we manipulated the temporal correlation between visual and auditory stimuli in a first-person fisherman video game. Subjects saw rapidly swimming fish whose size oscillated, either at 6 or 8 Hz. Subjects categorized each fish according to its rate of size oscillation, while trying to ignore a concurrent broadband sound seemingly emitted by the fish. In three experiments, categorization was faster and more accurate when the rate at which a fish oscillated in size matched the rate at which the accompanying, task-irrelevant sound was amplitude modulated. Control conditions showed that the difference between responses to matched and mismatched audiovisual signals reflected a performance gain in the matched condition, rather than a cost from the mismatched condition. The performance advantage with matched audiovisual signals was remarkably robust over changes in task demands between experiments. Performance with matched or unmatched audiovisual signals improved over successive trials at about the same rate, emblematic of perceptual learning in which visual oscillation rate becomes more discriminable with experience. Finally, analysis at the level of individual subjects' performance pointed to differences in the rates at which subjects can extract information from audiovisual stimuli.

Bi-directional audiovisual influences on temporal modulation discrimination

“…experiments, in which no rate effects were found. The differences between the previous results and the current set of results may be due to some combination of the different stimulus rates used in each experiment (6 and 7 Hz in the current study, vs 6 and 8 Hz in the previous studies) and the different trial blocking employed in each experiment (congruent, incongruent, and unmodulated trials were intermingled, in random order, in the current experiment, compared to having the three conditions in separate blocks of trials in Sun et al, 2016). Taken together, these results point to a need for additional experiments to examine interactions between cross-modal temporal modulations and the rates at which they occur.…”

“…Two recent studies of multisensory temporal processing examined the influence of dynamic auditory stimuli on judgments of visual modulation rates in a multisensory scene offered by a video game (Fish Police! ; Goldberg et al, 2015;Sun et al, 2016). Players were asked to judge the rate at which a visual stimulus (a computer-generated fish) was modulated in size while it moved across the display and emitted an intensity-modulated sound.…”

Modulated stimuli demonstrate asymmetric interactions between hearing and vision

“…The results of Experiment 1 showed that task-irrelevant visual information strongly affected performance on an auditory discrimination task. To better understand the mechanisms of auditory-visual interactions, in Experiment 2 we tested the prediction that such interactions require auditory and visual changes to occur in synchrony 20,21 . Importantly our auditory and visual stimuli allowed us to manipulate synchrony equivalently in both modalities in terms of the modulation rate of amplitude or size changes.…”

“…Continuously modulated auditory and visual stimuli have previously been used to study auditory-visual interactions 20,21 , but not to determine whether each modality exerts equal influence on the other. For example, Maddox et al .…”

Temporal Audiovisual Motion Prediction in 2D- vs. 3D-Environments

“…One reason for this variance in response patterns might be different rates at which participants can use audiovisual information. In recent experiments of Sun et al (2017) participants had to react to audiovisual oscillating fish stimuli and they differed in the rate at which they were able to extract information from audiovisual stimuli for this task. Law et al (1993) also demonstrated interindividual variability for visual time to arrival judgments and proposed that participants differ in their ability to integrate several visual information sources.…”