Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments

Li, Xiaofei; Ban, Yutong; Girin, Laurent; Alameda-Pineda, Xavier; Horaud, Radu

doi:10.1109/jstsp.2019.2903472

Cited by 39 publications

(44 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The present work is also related to previous work in localizing sounds in visual inputs [20,14,22,9,8,24,4,35], which aims to identify which pixels in a video are associated with an object making a particular sound.…”

Section: Sound Localizationmentioning

confidence: 97%

Self-Supervised Moving Vehicle Tracking With Stereo Sound

Gan

Zhao²,

Chen³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

159

102

View full text Add to dashboard Cite

Humans are able to localize objects in the environment using both visual and auditory cues, integrating information from multiple modalities into a common reference frame. We introduce a system that can leverage unlabeled audiovisual data to learn to localize objects (moving vehicles) in a visual reference frame, purely using stereo sound at inference time. Since it is labor-intensive to manually annotate the correspondences between audio and object bounding boxes, we achieve this goal by using the co-occurrence of visual and audio streams in unlabeled videos as a form of selfsupervision, without resorting to the collection of ground truth annotations. In particular, we propose a framework that consists of a vision "teacher" network and a stereosound "student" network. During training, knowledge embodied in a well-established visual vehicle detection model is transferred to the audio domain using unlabeled videos as a bridge. At test time, the stereo-sound student network can work independently to perform object localization using just stereo audio and camera meta-data, without any visual input. Experimental results on a newly collected Auditory Vehicle Tracking dataset verify that our proposed approach outperforms several baseline approaches. We also demonstrate that our cross-modal auditory localization approach can assist in the visual localization of moving vehicles under poor lighting conditions.

show abstract

Section: Sound Localizationmentioning

confidence: 97%

Self-Supervised Moving Vehicle Tracking With Stereo Sound

Gan

Zhao²,

Chen³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

159

102

View full text Add to dashboard Cite

show abstract

“…The proposed vM-VEM tracker yields the lowest false alarm (FA) rate of 5.9% and MAE of 2.6, and the second lowest MD rate of 23.9%. The GM-FO variant of [16] yields an MD rate of 22.3% since it uses velocity information to smooth the trajectories. This illustrates the advantage of the von-Mises distribution to model directional data (DOA).…”

Section: Experimental Evaluationmentioning

confidence: 99%

“…By using the fact that the prior distribution on z tm is denoted by p(z tm = n) = π n , we can now write the a posteriori distribution as q(z tm = n) ∝ π n β tmn with: β tmn = ω tm κ y A(ω tm κ y ) cos(y tm − µ tn ) n = 0 1/2π n = 0 , thus leading to the results in (16) and (3).…”

Section: Appendix B Derivation Of the E-z Stepmentioning

confidence: 99%

“…This paper builds on [7], [15], [16] and proposes to use the von Mises distribution to model the DOAs of multiple acoustic sources with circular random variables. The Bayesian filtering formulation for the multi-source tracking problem is intractable over time, due to the combinatorial nature of the unknown association between observed variables and latent variables.…”

Section: Introductionmentioning

confidence: 99%

“…Results obtained with recordings #1 (left) and #2 (right) from Task 6 of the LOCATA dataset. Top-to-down: vM-PHD[13], GM-FO[16], vM-VEM (proposed) and groundtruth trajectories. Different colors represent different audio sources.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Tracking Multiple Audio Sources With the von Mises Distribution and Variational EM

Ban

Alameda-Pineda

Evers

et al. 2019

IEEE Signal Process. Lett.

Self Cite

View full text Add to dashboard Cite

In this paper we address the problem of simultaneously tracking several moving audio sources, namely the problem of estimating source trajectories from a sequence of observed features. We propose to use the von Mises distribution to model audio-source directions of arrival with circular random variables. This leads to a Bayesian filtering formulation which is intractable because of the combinatorial explosion of associating observed variables with latent variables, over time. We propose a variational approximation of the filtering distribution. We infer a variational expectation-maximization algorithm that is both computationally tractable and time efficient. We propose an audio-source birth method that favors smooth source trajectories and which is used both to initialize the number of active sources and to detect new sources. We perform experiments with the recently released LOCATA dataset comprising two moving sources and a moving microphone array mounted onto a robot.

show abstract