Bidirectional audiovisual interactions: Evidence from a computerized fishing game

Abstract

We used a specially designed computer game to examine behavioral consequences of audiovisual integration. Target stimuli (animated fish swimming across the computer screen) were modulated in size and/or emitted an amplitude-modulated sound. Modulations, visual or auditory, were at 6 or 7 Hz (corresponding to “slow” and “fast”). In one game, subjects were instructed to categorize successive fish as “slow” or “fast” based on the auditory modulations; in another game, they categorized fish based on visual modulation rate. In each game, subjects were instructed to ignore input from the task-irrelevant modality. In each game, modulations could be (1) present only in the modality of interest, (2) present and matching in both modalities, or (3) present but mismatched between modalities. While reaction times were similar across games, accuracy was highest when auditory modulation was the basis for categorizing fish. Accuracy and reaction times improved when cross-modal modulation rates matched, and worsened when modulation rates conflicted. Additionally, accuracy was more strongly affected by between-modality congruence/incongruence when subjects attended to visual modulations than when they attended to auditory ones. Results indicate that audiovisual integration is not entirely under volitional control, and that competition between sensory modalities adversely impacts perception in dynamic environments.

Policing Fish at Boston's Museum of Science: Studying Audiovisual Interaction in the Wild

“…Planning this project made us mindful of the many differences between a public or quasi-public research environment and the well-controlled dedicated research laboratories in which we have been studying various aspects of audiovisual interactions ( Benussen et al., 2014 ; Sun et al., 2014 ). Anticipating that other researchers might want to carry out a project in a setting like the one we found at the Museum, we offer some suggestions for optimizing such opportunities.…”

“…This particular form of mismatch between auditory and visual signals was intended to maximize stimulus-response interference (Fitts & Deininger, 1954), which was expected to maximize observed differences between responses to Congruent and Incongruent fish. Planning this project made us mindful of the many differences between a public or quasipublic research environment and the well-controlled dedicated research laboratories in which we have been studying various aspects of audiovisual interactions (Benussen et al, 2014;Sun et al, 2014). Anticipating that other researchers might want to carry out a project in a setting like the one we found at the Museum, we offer some suggestions for optimizing such opportunities.…”

“…For some time, we have been interested in audiovisual integration, particularly the ways in which a correlation between signals' temporal structures promotes their integration ( Parise, Spence, & Ernst, 2012 ). So we took advantage of the Museum environment in order to examine age-related changes in audiovisual integration, and also to test the robustness of observations made previously, in well-controlled laboratory settings ( Benussen et al., 2014 ; Sun, Shinn-Cunningham, Somers, & Sekuler, 2014 ).…”