Relating Sound and Sight in Simulated Environments

Abstract

The auditory signals at the ear can be affected by components arriving both directly from a sound source and indirectly via environmental reverberation. Previous studies have suggested that the perceptual separation of these contributions can be aided by expectations of likely reverberant qualities. Here, we investigated whether the visual sense can provide information about the auditory properties of physical locations that could also be used to develop such expectations. We presented participants with audiovisual stimuli derived from 10 simulated real-world locations via a virtual reality (VR; n=44) or web-based (n=60) delivery method. On each trial, participants viewed a first-person perspective rendering of a location before hearing a spoken utterance that was convolved with an impulse response that was from a location that was either the same as (congruent) or different to (incongruent) the visually-depicted location. We find that the odds of participants reporting an audiovisual match was around 2.6 times higher for congruent trials than for incongruent trials. Additional analyses suggest that this effect was comparable for both VR and web-based delivery methods and showed heterogeneity across the set of 10 locations. Analysis of the incongruent trials suggests that participants were more likely to confuse audiovisual pairs as matching if their locations had similar reverberation times. Overall, this study suggests that human perceivers have a capacity to form expectations of reverberation from visual information. Such expectations may be useful for the perceptual challenge of separating sound sources and reverberation from within the signal available at the ear.