Audio-Visual Tracking of Concurrent Speakers

Qian, Xinyuan; Brutti, Alessio; Lanz, Oswald; Omologo, Maurizio; Cavallaro, Andrea

doi:10.1109/tmm.2021.3061800

Cited by 16 publications

(15 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the development of deep learning techniques, more and more research attention has been paid to how to combine/fuse audio and visual for vision-related tasks, e.g., sounding object localization [18], audio-visual synchronization [73], object tracking [74], and saliency detection [75]. Though the primary focus of this review is on saliency detection, we shall still review several most representative audiovisual-related tasks [76], [77], [78] in advance because these fusion-related arts can be directly referred to and get a deep insight into our audio-visual saliency detection. For a better reading, we propose to introduce three most representative tasks here, including audio-visual correspondence (AVC), face and audio matching (FAM) and sound-object localization (SOL).…”

Section: Audio-visual Multi-modality Fusionmentioning

confidence: 99%

A Comprehensive Survey on Video Saliency Detection With Auditory Information: The Audio-Visual Consistency Perceptual is the Key!

Chen

Mengke

Song

et al. 2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Video saliency detection (VSD) aims at fast locating the most attractive objects/things/patterns in a given video clip. Existing VSD-related works have mainly relied on the visual system but paid less attention to the audio aspect. In contrast, our audio system is the most vital complementary part of our visual system. Also, audio-visual saliency detection (AVSD), one of the most representative research topics for mimicking human perceptual mechanisms, is currently in its infancy, and none of the existing survey papers have touched on it, especially from the perspective of saliency detection. Thus, the ultimate goal of this paper is to provide an extensive review to bridge the gap between audio-visual fusion and saliency detection. In addition, as another highlight of this review, we have provided a deep insight into key factors that could directly determine AVSD deep models' performances. We claim that the audio-visual consistency degree (AVC) -a long-overlooked issue, can directly influence the effectiveness of using audio to benefit its visual counterpart when performing saliency detection. Moreover, to make the AVC issue more practical and valuable for future followers, we have newly equipped almost all existing publicly available AVSD datasets with additional frame-wise AVC labels. Based on these upgraded datasets, we have conducted extensive quantitative evaluations to ground our claim on the importance of AVC in the AVSD task. In a word, our ideas and new sets serve as a convenient platform with preliminaries and guidelines, all of which can potentially facilitate future works in further promoting stateof-the-art (SOTA) performance.

show abstract

Section: Audio-visual Multi-modality Fusionmentioning

confidence: 99%

A Comprehensive Survey on Video Saliency Detection With Auditory Information: The Audio-Visual Consistency Perceptual is the Key!

Chen

Mengke

Song

et al. 2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…To evaluate the TLR in 3D, a target is considered to be lost if the error with respect to the ground-truth is larger than 300 mm. We also use a fine error metric defined as 3D , where only the frames where tracking is successful are considered in Equation (27).…”

Section: Evaluation Metricsmentioning

confidence: 99%

Audiovisual Tracking of Multiple Speakers in Smart Spaces

Sanabria-Macias,

Marron-Romera,

Macias-Guarasa

2023

Sensors

View full text Add to dashboard Cite

This paper presents GAVT, a highly accurate audiovisual 3D tracking system based on particle filters and a probabilistic framework, employing a single camera and a microphone array. Our first contribution is a complex visual appearance model that accurately locates the speaker’s mouth. It transforms a Viola & Jones face detector classifier kernel into a likelihood estimator, leveraging knowledge from multiple classifiers trained for different face poses. Additionally, we propose a mechanism to handle occlusions based on the new likelihood’s dispersion. The audio localization proposal utilizes a probabilistic steered response power, representing cross-correlation functions as Gaussian mixture models. Moreover, to prevent tracker interference, we introduce a novel mechanism for associating Gaussians with speakers. The evaluation is carried out using the AV16.3 and CAV3D databases for Single- and Multiple-Object Tracking tasks (SOT and MOT, respectively). GAVT significantly improves the localization performance over audio-only and video-only modalities, with up to 50.3% average relative improvement in 3D when compared with the video-only modality. When compared to the state of the art, our audiovisual system achieves up to 69.7% average relative improvement for the SOT and MOT tasks in the AV16.3 dataset (2D comparison), and up to 18.1% average relative improvement in the MOT task for the CAV3D dataset (3D comparison).

show abstract

“…Active speaker detection is a multi-modal task aiming to identify active speakers from a set of candidates in an arbitrary video. This task plays an essential role in speaker diarization [7,41], speaker tracking [27,28], automatic video editing [10,19], and other applications, which has attracted extensive attention from both industry and academia. Figure 1.…”

Section: Introductionmentioning

confidence: 99%

A Light Weight Model for Active Speaker Detection

Liao¹,

Duan²,

Kanghui³

et al. 2023

Preprint

View full text Add to dashboard Cite

Active speaker detection is a challenging task in audiovisual scenario understanding, which aims to detect who is speaking in one or more speakers scenarios. This task has received extensive attention as it is crucial in applications such as speaker diarization, speaker tracking, and automatic video editing. The existing studies try to improve performance by inputting multiple candidate information and designing complex models. Although these methods achieved outstanding performance, their high consumption of memory and computational power make them difficult to be applied in resource-limited scenarios. Therefore, we construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%), while the resource costs are significantly lower than the state-of-the-art method, especially in model parameters (1.0M vs. 22.5M, about 23×) and FLOPs (0.6G vs. 2.6G, about 4×). In addition, our framework also performs well on the Columbia dataset showing good robustness. The code and model weights are available at https: //github.com/Junhua-Liao/Light-ASD.

show abstract

Audio-Visual Tracking of Concurrent Speakers

Cited by 16 publications

References 58 publications

A Comprehensive Survey on Video Saliency Detection With Auditory Information: The Audio-Visual Consistency Perceptual is the Key!

A Comprehensive Survey on Video Saliency Detection With Auditory Information: The Audio-Visual Consistency Perceptual is the Key!

Audiovisual Tracking of Multiple Speakers in Smart Spaces

A Light Weight Model for Active Speaker Detection

Contact Info

Product

Resources

About