Multimodal Contrastive Training for Visual Representation Learning

Yuan, Xin; Lin, Zhe; Kuen, Jason; Zhang, Jianming; Wang, Yilin; Maire, Michael; Kale, Ajinkya; Faieta, Baldo

doi:10.1109/cvpr46437.2021.00692

Cited by 98 publications

(44 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ICMLM [48] and VirTex [15] showed that language supervision on COCO Captions produced useful visual representations. Prior to CLIP, Multimodal Contrastive Training [62] adds contrastive imageimage and language-image losses to VirTex which further improve performance. CLIP [45] quickly garnered significant attention for its simplicity, scale, and strong results.…”

Section: Related Workmentioning

confidence: 99%

SLIP: Self-supervision meets Language-Image Pre-training

Mu¹,

Kirillov²,

Wagner³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent work has shown that self-supervised pre-training leads to improvements over supervised learning on challenging visual recognition tasks. CLIP, an exciting new approach to learning with language supervision, demonstrates promising performance on a wide variety of benchmarks. In this work, we explore whether self-supervised learning can aid in the use of language supervision for visual representation learning. We introduce SLIP, a multi-task learning framework for combining self-supervised learning and CLIP pre-training. After pre-training with Vision Transformers, we thoroughly evaluate representation quality and compare performance to both CLIP and self-supervised learning under three distinct settings: zero-shot transfer, linear classification, and end-to-end finetuning. Across Im-ageNet and a battery of additional datasets, we find that SLIP improves accuracy by a large margin. We validate our results further with experiments on different model sizes, training schedules, and pre-training datasets. Our findings show that SLIP enjoys the best of both worlds: better performance than self-supervision (+8.1% linear accuracy) and language supervision (+5.2% zero-shot accuracy).

show abstract

Section: Related Workmentioning

confidence: 99%

SLIP: Self-supervision meets Language-Image Pre-training

Mu¹,

Kirillov²,

Wagner³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Co-teaching+ (Yu et al, 2019) combines the features of both, in which each model teaches each other only on disagreement data. Yang et al (2021) propose mutual contrastive learning that enables two networks to learn extra contrastive information from each other. As text data is discrete and compositional, qualities of multiple augmentations can be uneven, which may corrupt the generalization of sentence embeddings.…”

Section: Related Workmentioning

confidence: 99%

PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings

Wu¹,

Tao²,

Shen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semanticallyclose positive instance to construct contrastive pairs. Nonetheless, existing approaches usually depend on a mono-augmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings. A straightforward solution is resorting to more diverse positives from a multiaugmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field. As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL can perform peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations. Experiments on STS benchmarks verify the effectiveness of our PCL against its competitors in unsupervised sentence embeddings.

show abstract

“…Specifically, [68][69][70][71][72] employ pretrained language models and object detectors to learn visual features well aligned with the embeddings of caption words. Recent works [73,74] improve training efficiency by removing the need for object detectors and scale to hundreds of millions of samples for substantial performance gains [75]. Moreover, [28] proposes a novel open-vocabulary learning task and shows that pretrained visual features improve not only the detection performance on base classes but also novel classes.…”

Section: Related Workmentioning

confidence: 99%

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Huynh¹,

Kuen²,

Lin³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.

show abstract

Multimodal Contrastive Training for Visual Representation Learning

Cited by 98 publications

References 29 publications

SLIP: Self-supervision meets Language-Image Pre-training

SLIP: Self-supervision meets Language-Image Pre-training

PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings

Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling

Contact Info

Product

Resources

About