3D audio-visual speaker tracking with an adaptive particle filter

Abstract

We propose an audiovisual fusion algorithm for 3D speaker tracking from a localised multi-modal sensor platform composed of a camera and a small microphone array. After extracting audiovisual cues from individual modalities we fuse them adaptively using their reliability in a particle filter framework. The reliability of the audio signal is measured based on the maximum Global Coherence Field (GCF) peak value at each frame. The visual reliability is based on colour-histogram matching with detection results compared with a reference image in the RGB space. Experiments on the AV16.3 dataset show that the proposed adaptive audiovisual tracker outperforms both the individual modalities and a classical approach with fixed parameters in terms of tracking accuracy.

Audiovisual Speaker Tracking Using Nonlinear Dynamical Systems With Dynamic Stream Weights

“…This corresponds to a small process noise, which cannot be handled efficiently using PFs. However, the PF baseline method from [32] shows a performance comparable to the EKF-based methods, which indicates that the MDF approach used in this algorithm is efficient on this dataset.…”

“…A comparison of the Bayesian filtering framework proposed in this study with state-of-the-art audiovisual speaker tracking methods is the primary focus of the second evaluation scenario. Four different frameworks were selected as baseline methods: the standard EKF with audiovisual observations, the audiovisual fusion technique based on an iterated EKF as proposed by Gehring et al [30], the PF-based approach with adaptive particle weighting introduced by Gerlach et al [31] and the recently proposed framework by Qian et al [32], which explicitly incorporates sensor reliability measures into the weighting stage of the PF. These methods are compared with the ODSW-EKF with Dirichlet prior and a DSW-EKF with corresponding prediction model based on the logistic function, as introduced in Sec.…”

“…The framework provided explicit control over the individual contributions of acoustic and visual observations via exponential weighting parameters, which were determined a-priori using a grid-search. A recently proposed algorithm for speaker tracking has explicitly considered sensor reliability measures within a particle filtering framework [32]. This work utilized the peak value of the acoustic global coherence field and the correlation between a color-histogram template and the detected face as features, which affected the weighting and resampling step of the particle filter.…”

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

“…More recently, audio-visual trackers based on particle filtering (PF) and probability hypothesis density (PHD) filters were proposed, e.g. [4]- [7], [20]- [22]. In [6] DOAs of audio sources to guide the propagation of particles and combined the filter with a mean-shift algorithm to reduce the computational complexity.…”

Exploiting the Complementarity of Audio and Visual Data in Multi-speaker Tracking

“…Alternatively, [7] used a Markov chain Monte Carlo particle filter (MCMC-PF) to increase sampling efficiency. Still in a particle filter tracking framework, [8] proposed to use the maximum global coherence field of the audio signal and image colorhistogram matching to adapt the reliability of audio and visual information. Finally, along a different line, [9] used visual tracking information to assist source separation and beamforming.…”