Audio-Visual Variational Fusion for Multi-Person Tracking with Robots

Alameda-Pineda, Xavier; Arias, Soraya; Ban, Yutong; Delorme, Guillaume; Girin, Laurent; Horaud, Radu; Li, Xiaofei; Morgue, Bastien; Sarrazin, Guillaume

doi:10.1145/3343031.3350590

Cited by 4 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, track initialization (birth) and de-activation (death) are required. Strategies usually rely on the temporal persistence of new observations and tracks, such as [13,46,47]. Within a given time interval, a new track is initialized if consistent un-associated observations appear in a nearby region.…”

Section: Introductionmentioning

confidence: 99%

Audio-Visual Tracking of Concurrent Speakers

Qian

Brutti

Lanz

et al. 2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Audio-visual tracking of an unknown number of concurrent speakers in 3D is a challenging task, especially when sound and video are collected with a compact sensing platform. In this paper, we propose a tracker that builds on generative and discriminative audio-visual likelihood models formulated in a particle filtering framework. We localize multiple concurrent speakers with a de-emphasized acoustic map assisted by the image detection-derived 3D video observations. The 3D multimodal observations are either assigned to existing tracks for discriminative likelihood computation or used to initialize new tracks. The generative likelihoods rely on color distribution of the target and the de-emphasized acoustic map. Experiments on AV16.3 and CAV3D datasets show that the proposed tracker outperforms the uni-modal trackers and the state-of-the-art approaches both in 3D and on the image plane.

show abstract

Section: Introductionmentioning

confidence: 99%

Audio-Visual Tracking of Concurrent Speakers

Qian

Brutti

Lanz

et al. 2022

IEEE Trans. Multimedia

View full text Add to dashboard Cite

show abstract

“…Recent works in the domain of audiovisual speaker localization showed promising results utilizing different data fusion strategies. For example, [5,6] introduce a variational Bayesian approximation to optimally merge acoustic and visual data for combined localization and tracking. A related approach introduced in [7] uses an expectation maximization (EM) algorithm for weighted clustering in the audiovisual observation space.…”

Section: Introductionmentioning

confidence: 99%

Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain

Wissing

Boenninghoff

Kolossa

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Estimating the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. Both applications benefit from a known speaker position when, for instance, applying beamforming or assigning unique speaker identities. Recently, several approaches utilizing acoustic signals augmented with visual data have been proposed for this task. However, both the acoustic and the visual modality may be corrupted in specific spatial regions, for instance due to poor lighting conditions or to the presence of background noise. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions in the localization space. This fusion is achieved via a neural network, which combines the predictions of individual audio and video trackers based on their time-and location-dependent reliability. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.

show abstract

“…These advances are fueled by the introduction of new large-scale video datasets such as Moments in Time [42], Kinetics [32], and ActivityNet [28]. The ability to automatically recognize activities and events is an important component in video surveillance [1,43,46], virtual/augmented reality [5,24], and human-robot interaction [3,25,41]. Here, we aim to improve activity recognition by incorporating information that is missed or inherently invisible purely from videos.…”

Section: Introductionmentioning

confidence: 99%

Heterogeneous Non-Local Fusion for Multimodal Activity Recognition

Byvshev

Mettes

Xiao

2020

Proceedings of the 2020 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

In this work, we investigate activity recognition using multimodal inputs from heterogeneous sensors. Activity recognition is commonly tackled from a single-modal perspective using videos. In case multiple signals are used, they come from the same homogeneous modality, e.g. in the case of color and optical flow. Here, we propose an activity network that fuses multimodal inputs coming from completely different and heterogeneous sensors. We frame such a heterogeneous fusion as a non-local operation. The observation is that in a non-local operation, only the channel dimensions need to match. In the network, heterogeneous inputs are fused, while maintaining the shapes and dimensionalities that fit each input. We outline both asymmetric fusion, where one modality serves to enforce the other, and symmetric fusion variants. To further promote research into multimodal activity recognition, we introduce GloVid, a first-person activity dataset captured with video recordings and smart glove sensor readings. Experiments on GloVid show the potential of heterogeneous non-local fusion for activity recognition, outperforming individual modalities and standard fusion techniques. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding.

show abstract

Audio-Visual Variational Fusion for Multi-Person Tracking with Robots

Cited by 4 publications

References 10 publications

Audio-Visual Tracking of Concurrent Speakers

Audio-Visual Tracking of Concurrent Speakers

Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain

Heterogeneous Non-Local Fusion for Multimodal Activity Recognition

Contact Info

Product

Resources

About