Policing Fish at Boston's Museum of Science: Studying Audiovisual Interaction in the Wild

Abstract

Boston's Museum of Science supports researchers whose projects advance science and provide educational opportunities to the Museum's visitors. For our project, 60 visitors to the Museum played “Fish Police!!,” a video game that examines audiovisual integration, including the ability to ignore irrelevant sensory information. Players, who ranged in age from 6 to 82 years, made speeded responses to computer-generated fish that swam rapidly across a tablet display. Responses were to be based solely on the rate (6 or 8 Hz) at which a fish's size modulated, sinusoidally growing and shrinking. Accompanying each fish was a task-irrelevant broadband sound, amplitude modulated at either 6 or 8 Hz. The rates of visual and auditory modulation were either Congruent (both 6 Hz or 8 Hz) or Incongruent (6 and 8 or 8 and 6 Hz). Despite being instructed to ignore the sound, players of all ages responded more accurately and faster when a fish's auditory and visual signatures were Congruent. In a controlled laboratory setting, a related task produced comparable results, demonstrating the robustness of the audiovisual interaction reported here. Some suggestions are made for conducting research in public settings.

Decision-Making and Multisensory Combination Under Time Stress

“…This smaller effect may have been caused by the relatively large difference between the oscillation frequencies used here (5 vs. 8 Hz). That large difference probably made the task easier than it had been with the 6 versus 8 Hz pair used by Goldberg et al. (2015).…”

“…Note that the effect of congruence shown in the upper panel of Figure 1 is considerably smaller than the comparable effect from some previous studies with Fish Police!! (e.g., Goldberg et al., 2015). This smaller effect may have been caused by the relatively large difference between the oscillation frequencies used here (5 vs. 8 Hz).…”

“…Previous multisensory studies from our laboratory used a bespoke video game, Fish Police!! to examine audiovisual combination under conditions that demanded flexible cognitive control (Goldberg et al., 2015; Sun et al., 2017; Varghese et al., 2017). Specifically, the game challenged subjects to switch without warning between (a) conditions whose auditory and visual signals contradicted with one another and therefore should not be combined, and (b) conditions whose auditory and visual signals were consistent with one another (Deroy et al., 2016).…”

Bi-directional audiovisual influences on temporal modulation discrimination

“…Two recent studies of multisensory temporal processing examined the influence of dynamic auditory stimuli on judgments of visual modulation rates in a multisensory scene offered by a video game (Fish Police! ; Goldberg et al, 2015;Sun et al, 2016). Players were asked to judge the rate at which a visual stimulus (a computer-generated fish) was modulated in size while it moved across the display and emitted an intensity-modulated sound.…”

Memory and learning for visual signals in time and space

“…These links include a suggestion that information from the two dimensions of processing converges at some site in the parietal lobe (e.g., Walsh 2003;Oliveri, Koch, & Caltagirone 2009). Such convergence might support a combination or coordination of temporal and spatial streams of information, as some psychophysical studies have shown (Goldberg et al, 2015;Keller & Sekuler, 2015). Although the tasks in those studies differed from ours, their results do suggest the possibility that by making both sources of information available at the same time, concurrent spatial and temporal presentations could enhance performance over what would be produced by either source alone.…”