Static Visual Spatial Priors for DoA Estimation

Abstract

As we interact with the world, for example when we communicate with our colleagues in a large open space or meeting room, we continuously analyse the surrounding environment and, in particular, localise and recognise acoustic events. While we largely take such abilities for granted, they represent a challenging problem for current robots or smart voice assistants as they can be easily fooled by high degree of sound interference in acoustically complex environments. Preventing such failures when using solely audio data is challenging, if not impossible since the algorithms need to take into account wider context and often understand the scene on a semantic level. In this paper, we propose what to our knowledge is the first multi-modal direction of arrival (DoA) of sound, which uses static visual spatial prior providing an auxiliary information about the environment to suppress some of the false DoA detections. We validate our approach on a newly collected real-world dataset, and show that our approach consistently improves over classic DoA baselines.

Building Proactive Voice Assistants: When and How (not) to Interact

“…5 B). When presence of a user is detected and the device has some update ready, or a voicetrigger is spotted, the device wakes up and faces the user [47], starting to process at the same time the audio-visual data. This, depending on the compute requirements can happen either on device or in the cloud (Fig.…”

“…Additionally, we estimate direction of arrival (DOA) θ s for each of the detected sounds ′ s ′ using a set of DOA estimates from the raw signal (as many as detected acoustic events at each given time step), which are then mapped to x, y, z coordinates 3 . This process can leverage an additional semantic information from vision stream, as shown in [47]. The most likely pairs {acoustic_event, θ s } for co-occurring events are estimated in the spatial model using visual data.…”