Object-aware Contrastive Learning for Debiased Scene Representation

Abstract

Contrastive self-supervised learning has shown impressive results in learning visual representations from unlabeled images by enforcing invariance against different data augmentations. However, the learned representations are often contextually biased to the spurious scene correlations of different objects or object and background, which may harm their generalization on the downstream tasks. To tackle the issue, we develop a novel object-aware contrastive learning framework that first (a) localizes objects in a self-supervised manner and then (b) debias scene correlations via appropriate data augmentations considering the inferred object locations. For (a), we propose the contrastive class activation map (ContraCAM), which finds the most discriminative regions (e.g., objects) in the image compared to the other images using the contrastively trained models. We further improve the ContraCAM to detect multiple objects and entire shapes via an iterative refinement procedure. For (b), we introduce two data augmentations based on ContraCAM, object-aware random crop and background mixup, which reduce contextual and background biases during contrastive self-supervised learning, respectively. Our experiments demonstrate the effectiveness of our representation learning framework, particularly when trained under multi-object images or evaluated under the background (and distribution) shifted images. 1 * Equal contribution. 1 Code is available at https://github.com/alinlab/occon. 2 Some recent works (e.g., [14,15]) attract the positives without contrasting the negatives. While we mainly focus on contrastive learning with negatives, our method is also applicable to the positive-only methods.Preprint. Under review.

Effective Multi-Object Tracking via Global Object Models and Object Constraint Learning

“…Some contrasting learning methods [ 41 , 42 , 43 ] are proposed to enhance self-supervised learning performance with the contrastive learning [ 42 ] aims to tackle background the bias problem in contrastive learning. Ref.…”

Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition

“…In this aspect, various benchmarks have been proposed to measure the robustness under distribution shifts [9,14,23,25,26,29,30,45,48,50], and this problem has been extensively studied in broad research fields [3,4,10,15,16,24,38,39,40,43,52,55,62]. Among them, benchmarking robustness [23] and resolving scene bias [10,42] or distribution shift [43,59] are the most related to our problem setup. Different from the aforementioned works, we first explore the background shift issue in the CSLR task with a newly synthesized benchmark.…”