With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

Dwibedi, Debidatta; Aytar, Yusuf; Tompson, Jonathan; Sermanet, Pierre; Zisserman, Andrew

doi:10.1109/iccv48922.2021.00945

Cited by 260 publications

(149 citation statements)

References 29 publications

Supporting

Mentioning

118

Contrasting

Order By: Relevance

“…Contrastive Learning has shown remarkable advantages in self-supervised learning [6,11,15,18,45,48]. The contrastive loss measures the similarity of representation pairs and attempts to distinguish between positive and negative pairs.…”

Section: Contrastive Representation Learningmentioning

confidence: 99%

Low-confidence Samples Matter for Domain Adaptation

Zhang¹,

Li²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

Domain adaptation (DA) aims to transfer knowledge from a label-rich source domain to a related but labelscarce target domain. The conventional DA strategy is to align the feature distributions of the two domains. Recently, increasing researches have focused on self-training or other semi-supervised algorithms to explore the data structure of the target domain. However, the bulk of them depend largely on confident samples in order to build reliable pseudo labels, prototypes or cluster centers. Representing the target data structure in such a way would overlook the huge low-confidence samples, resulting in suboptimal transferability that is biased towards the samples similar to the source domain. To overcome this issue, we propose a novel contrastive learning method by processing low-confidence samples, which encourages the model to make use of the target data structure through the instance discrimination process. To be specific, we create positive and negative pairs only using low-confidence samples, and then re-represent the original features with the classifier weights rather than directly utilizing them, which can better encode the task-specific semantic information. Furthermore, we combine cross-domain mixup to augment the proposed contrastive loss. Consequently, the domain gap can be well bridged through contrastive learning of intermediate representations across domains. We evaluate the proposed method in both unsupervised and semi-supervised DA settings, and extensive experimental results on benchmarks reveal that our method is effective and achieves state-ofthe-art performance. The code can be found in https: //github.com/zhyx12/MixLRCo.

show abstract

Section: Contrastive Representation Learningmentioning

confidence: 99%

Low-confidence Samples Matter for Domain Adaptation

Zhang¹,

Li²,

Wang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The key idea of instance discrimination is to treat each instance as a single category. Aside from generating positive samples from a single instance, other approaches assign samples from the same cluster or nearest neighbours as positives [5,10,18]. In order to eliminate the requirement of the massive number of negative samples, BYOL [14] and SimSiam [9] achieve competitive performance without any negative instance.…”

Section: Related Workmentioning

confidence: 99%

What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?

Qian¹,

Tian²,

Chen³

2022

Preprint

View full text Add to dashboard Cite

Self-supervised learning establishes a new paradigm of learning representations with much fewer or even no label annotations. Recently there has been remarkable progress on large-scale contrastive learning models which require substantial computing resources, yet such models are not practically optimal for small-scale tasks. To fill the gap, we aim to study contrastive learning on the wearablebased activity recognition task. Specifically, we conduct an in-depth study of contrastive learning from both algorithmic-level and tasklevel perspectives. For algorithmic-level analysis, we decompose contrastive models into several key components and conduct rigorous experimental evaluations to better understand the efficacy and rationale behind contrastive learning. More importantly, for task-level analysis, we show that the wearable-based signals bring unique challenges and opportunities to existing contrastive models, which cannot be readily solved by existing algorithms. Our thorough empirical studies suggest important practices and shed light on future research challenges. In the meantime, this paper presents an open-source PyTorch library CL-HAR, which can serve as a practical tool for researchers 1 . The library is highly modularized and easy to use, which opens up avenues for exploring novel contrastive models quickly in the future. CCS CONCEPTS• Computing methodologies → Unsupervised learning; Learning settings; Neural networks; • Human-centered computing → Ubiquitous and mobile computing.

show abstract

“…Among them, BarlowTwins [57] considers an objective function measuring the cross-correlation matrix between the features, and VicReg [3] uses a mix of variance, invariance and covariance regularizations. Methods such as [19] have explored the use of nearest-neighbour retrieval and divide and conquer [52]. However, none of these works studied the ability of SSL methods to learn continually and adaptively.…”

Section: Related Workmentioning

confidence: 99%

“…Self-Supervised Learning. The training procedure of several state-of-the-art SSL methods [3,7,8,13,19,26,28,57] can be summarized as follows. Given an image x in a batch sampled from a distribution D, two correlated views x A and x B are extracted by applying stochastic image augmentations, such as random cropping, color jittering and horizontal flipping.…”

Section: Preliminariesmentioning

confidence: 99%

Self-Supervised Models are Continual Learners

Fini¹,

Costa²,

Alameda-Pineda³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophically reduced in a Continual Learning (CL) scenario where data is presented to the model sequentially. In this paper, we show that self-supervised loss functions can be seamlessly converted into distillation mechanisms for CL by adding a predictor network that maps the current state of the representations to their past state. This enables us to devise a framework for Continual self-supervised visual representation Learning that (i) significantly improves the quality of the learned representations, (ii) is compatible with several state-of-the-art self-supervised objectives, and (iii) needs little to no hyperparameter tuning. We demonstrate the effectiveness of our approach empirically by training six popular self-supervised models in various CL settings.

show abstract

With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

Cited by 260 publications

References 29 publications

Low-confidence Samples Matter for Domain Adaptation

Low-confidence Samples Matter for Domain Adaptation

What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?

Self-Supervised Models are Continual Learners

Contact Info

Product

Resources

About