How visual cues help us understand speech in a complex environment.

Abstract

At a cocktail party, visual cues may help a listener by showing them where or when to direct attention, what acoustic modulations a target utterance contains, and/or what articulatory gestures produce the target. Here, we investigated target speech intelligibility while varying the visual cues available in a complex, confusing auditory scene. In all cases, subjects listened for a target utterance in the presence of multiple masker utterances with similar grammatical structure spoken by the same talker. The timing and direction of the target (and maskers) varied randomly, increasing the uncertainty about where and when to focus auditory attention. The number of correctly reported target key words measured performance. Performance tended to improve as the amount of visual information increased, particularly when masker phrases came from the direction of the target. Performance was generally similar whether listeners saw full videos of the target talker from the correct direction or only a static image of the talker at the right time in the correct direction. However, temporal information about where and when the target occurred improved performance over knowing only target location. Results suggest that in these scenes, visual cues aid target understanding by indicating where and roughly when to direct attention.