Self-supervised object detection from audio-visual correspondence

Afouras, Triantafyllos; Asano, Yuki; Fagan, Francois; Vedaldi, Andrea; Metze, Florian

doi:10.48550/arxiv.2104.06401

Cited by 7 publications

(9 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the comparison of the classifier CAM and ContraCAM, we use the publicly available supervised classifier 8 and MoCov2 9 trained on the ImageNet dataset under the ResNet-50 architecture. Here, we do not apply the expansion trick and run a single iteration for the ContraCAM.…”

Section: B1 Implementation Details For Object Localization Resultsmentioning

confidence: 99%

“…Self-supervised learning of visual representations from unlabeled images is a fundamental task of machine learning, which establishes various applications including object recognition [1,2], reinforcement learning [3,4], out-of-distribution detection [5,6], and multimodal learning [7,8]. Recently, contrastive learning [1,2,[9][10][11][12][13][14][15] has shown remarkable advances along this line.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Object-aware Contrastive Learning for Debiased Scene Representation

Mo¹,

Kang²,

Sohn³

et al. 2021

Preprint

View full text Add to dashboard Cite

Contrastive self-supervised learning has shown impressive results in learning visual representations from unlabeled images by enforcing invariance against different data augmentations. However, the learned representations are often contextually biased to the spurious scene correlations of different objects or object and background, which may harm their generalization on the downstream tasks. To tackle the issue, we develop a novel object-aware contrastive learning framework that first (a) localizes objects in a self-supervised manner and then (b) debias scene correlations via appropriate data augmentations considering the inferred object locations. For (a), we propose the contrastive class activation map (ContraCAM), which finds the most discriminative regions (e.g., objects) in the image compared to the other images using the contrastively trained models. We further improve the ContraCAM to detect multiple objects and entire shapes via an iterative refinement procedure. For (b), we introduce two data augmentations based on ContraCAM, object-aware random crop and background mixup, which reduce contextual and background biases during contrastive self-supervised learning, respectively. Our experiments demonstrate the effectiveness of our representation learning framework, particularly when trained under multi-object images or evaluated under the background (and distribution) shifted images. 1 * Equal contribution. 1 Code is available at https://github.com/alinlab/occon. 2 Some recent works (e.g., [14,15]) attract the positives without contrasting the negatives. While we mainly focus on contrastive learning with negatives, our method is also applicable to the positive-only methods.Preprint. Under review.

show abstract

Section: B1 Implementation Details For Object Localization Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Object-aware Contrastive Learning for Debiased Scene Representation

Mo¹,

Kang²,

Sohn³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The advancement of Deep Learning enabled a multitude of self-supervised approaches for localizing sounds in recent years. The line of work most relevant for the problem we are considering in this manuscript aims at locating sound sources in unlabeled videos [2], [3], [6], [9], [12], [13]. Arandjelović and Zisserman [3] propose a framework for cross-modal selfsupervision from video, enabling the localization of soundemitting objects by correlating the features produced by an audio-specific and image-specific encoding network.…”

Section: Related Workmentioning

confidence: 99%

“…Self-supervised approaches for audio-visual object detection, in contrast, are able to localize sound-emitting objects in videos without explicit manual annotations for object positions. The majority of these approaches are based on audiovisual co-occurrence of shared features in both domains [2], [6]. The main idea in these works is to contrast two random video frames and their corresponding audio segment from a large-scale dataset with each other, leveraging the fact that the chance of randomly sampled frames from different videos containing the same object type is negligible.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Moving Vehicle Detection from Audio-Visual Cues

Zürn¹,

Burgard²

2022

Preprint

View full text Add to dashboard Cite

Robust detection of moving vehicles is a critical task for any autonomously operating outdoor robot or selfdriving vehicle. Most modern approaches for solving this task rely on training image-based detectors using large-scale vehicle detection datasets such as nuScenes or the Waymo Open Dataset. Providing manual annotations is an expensive and laborious exercise that does not scale well in practice. To tackle this problem, we propose a self-supervised approach that leverages audio-visual cues to detect moving vehicles in videos. Our approach employs contrastive learning for localizing vehicles in images from corresponding pairs of images and recorded audio.In extensive experiments carried out with a real-world dataset, we demonstrate that our approach provides accurate detections of moving vehicles and does not require manual annotations. We furthermore show that our model can be used as a teacher to supervise an audio-only detection model. This student model is invariant to illumination changes and thus effectively bridges the domain gap inherent to models leveraging exclusively vision as the predominant modality.

show abstract

“…However, the perception process is usually abstract, making it difficult to manually label quantitative tags. Due to the natural correspondence between sound and vision, necessary supervision is provided for audio-visual learning (Hu et al 2020) (Afouras et al 2021). Therefore, we design a cross-modal self-supervised learning method, which exploits the complementarity and consistency of multi-modal data to generate weight labels of perception.…”

Section: Introductionmentioning

confidence: 99%

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Liu

Tang

2022

AAAI

View full text Add to dashboard Cite

Multi-modal fusion is proven to be an effective method to improve the accuracy and robustness of speaker tracking, especially in complex scenarios. However, how to combine the heterogeneous information and exploit the complementarity of multi-modal signals remains a challenging issue. In this paper, we propose a novel Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Specifically, a novel acoustic map based on spatial-temporal Global Coherence Field (stGCF) is first constructed for heterogeneous signal fusion, which employs a camera model to map audio cues to the localization space consistent with the visual cues. Then a multi-modal perception attention network is introduced to derive the perception weights that measure the reliability and effectiveness of intermittent audio and video streams disturbed by noise. Moreover, a unique cross-modal self-supervised learning method is presented to model the confidence of audio and visual observations by leveraging the complementarity and consistency between different modalities. Experimental results show that the proposed MPT achieves 98.6% and 78.3% tracking accuracy on the standard and occluded datasets, respectively, which demonstrates its robustness under adverse conditions and outperforms the current state-of-the-art methods.

show abstract

Self-supervised object detection from audio-visual correspondence

Cited by 7 publications

References 72 publications

Object-aware Contrastive Learning for Debiased Scene Representation

Object-aware Contrastive Learning for Debiased Scene Representation

Self-Supervised Moving Vehicle Detection from Audio-Visual Cues

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Contact Info

Product

Resources

About