With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

Dwibedi, Debidatta; Aytar, Yusuf; Tompson, Jonathan; Sermanet, Pierre; Zisserman, Andrew

doi:10.48550/arxiv.2104.14548

Cited by 20 publications

(30 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nearest-neighbor supervision Recently, researchers have exploited nearest-neighbor supervision to learn visual features (Dwibedi et al, 2021;Van Gansbeke et al, 2021). They find that using nearest-neighbor as positive samples in the contrastive loss improves the performances on multiple downstream tasks.…”

Section: Supervisionmentioning

confidence: 99%

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Li¹,

Liang²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, large-scale Contrastive Language-Image Pre-training (CLIP) (Radford et al., 2021) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training, thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our DeCLIP can learn generic visual features more efficiently. Instead of using the single image-text contrastive supervision, we fully exploit data potential through the use of (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighbor supervision from other similar pairs. Benefiting from these intrinsic supervision, our DeCLIP-ResNet50 can achieve 60.4% zero-shot top1 accuracy on ImageNet, which is 0.8% above the CLIP-ResNet50 while using 7.1× fewer data. Our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets when transferred to downstream tasks. Moreover, Scaling up the model and computing also works well in our framework. Our code, dataset and models are released at: https://github.com/Sense-GVT/ * The first three authors contribute equally. The order is determined by dice rolling.

show abstract

Section: Supervisionmentioning

confidence: 99%

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Li¹,

Liang²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…This methodology has shown great promise in learning visual representations without annotation [2,23,29,30,36,43,47]. More recently, contrastive methods based on the Siamese structure achieve remarkable performance on downstream tasks [5,7,8,15,17,20,40,45,46], some of which even surpass supervised models.…”

Section: Contrastive Learningmentioning

confidence: 99%

Crafting Better Contrastive Views for Siamese Representation Learning

Peng¹,

Wang²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent self-supervised contrastive learning methods greatly benefit from the Siamese structure that aims at minimizing distances between positive pairs. For high performance Siamese representation learning, one of the keys is to design good contrastive pairs. Most previous works simply apply random sampling to make different crops of the same image, which overlooks the semantic information that may degrade the quality of views. In this work, we propose ContrastiveCrop, which could effectively generate better crops for Siamese representation learning. Firstly, a semantic-aware object localization strategy is proposed within the training process in a fully unsupervised manner. This guides us to generate contrastive views which could avoid most false positives (i.e., object vs. background). Moreover, we empirically find that views with similar appearances are trivial for the Siamese model training. Thus, a center-suppressed sampling is further designed to enlarge the variance of crops. Remarkably, our method takes a careful consideration of positive pairs for contrastive learning with negligible extra training overhead. As a plugand-play and framework-agnostic module, ContrastiveCrop consistently improves SimCLR, MoCo, BYOL, SimSiam by 0.4% ∼ 2.0% classification accuracy on CIFAR-10, CIFAR-100, Tiny ImageNet and STL-10. Superior results are also achieved on downstream detection and segmentation tasks when pre-trained on ImageNet-1K.

show abstract

“…However, the neighbors had to be computed off-line at fixed intervals during training. Concurrent to our work, Dwibedi et al [17] adopted nearest neighbors from a memory bank under the BYOL [20] framework. The authors focus on image classification datasets.…”

Section: Related Workmentioning

confidence: 99%

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations

Gansbeke¹,

Vandenhende²,

Georgoulis³

et al. 2021

Preprint

View full text Add to dashboard Cite

Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection. However, current methods are still primarily applied to curated datasets like ImageNet. In this paper, we first study how biases in the dataset affect existing methods. Our results show that an approach like MoCo [22] works surprisingly well across: (i) object-versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets. Second, given the generality of the approach, we try to realize further gains with minor modifications. We show that learning additional invariances -through the use of multi-scale cropping, stronger augmentations and nearest neighbors -improves the representations. Finally, we observe that MoCo learns spatially structured representations when trained with a multi-crop strategy. The representations can be used for semantic segment retrieval and video instance segmentation without finetuning. Moreover, the results are on par with specialized models. We hope this work will serve as a useful study for other researchers.

show abstract

With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

Cited by 20 publications

References 47 publications

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Crafting Better Contrastive Views for Siamese Representation Learning

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations

Contact Info

Product

Resources

About