Auditory spatial analysis in reverberant audio-visual multi-talker environments with congruent and incongruent visual room information

Abstract

In multi-talker situation, listeners have the challenge to identify a target speech source out of a mixture of interfering background noises. In the current study it was investigate how listeners analyze audio-visual scenes with varying complexity in terms of number of talkers and reverberation. Furthermore, the visual information of the room was either coherent with the acoustic room or incoherent. The listeners’ task was to locate an ongoing speech source in a mixture of other speech sources. The 3D audio-visual scenarios were presented using a loudspeaker array and virtual reality glasses. It was shown that room reverberation as well as the number of talkers in a scene influence the ability to analyze an auditory scene in terms of accuracy and response time. Incongruent visual information of the room did not affect this ability. When few talkers were presented simultaneously, listeners were able to quickly and accurately detect a target talker even in adverse room acoustical conditions. Reverberation started to affect the response time when four or more talkers were presented. The number of talkers became a significant factor for five or more simultaneous talkers.