Are Large-scale Datasets Necessary for Self-Supervised Pre-training?

El-Nouby, Alaaeldin; Izacard, Gautier; Touvron, Hugo; Laptev, Ivan; Grave, Édouard

doi:10.48550/arxiv.2112.10740

Cited by 28 publications

(40 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some recent approaches have started to explore the combination of joint-embedding architectures and denoising pre-training tasks (El-Nouby et al, 2021;Baevski et al, 2022;Zhou et al, 2021). Those approaches mask an image by replacing the masked patches with a learnable mask token, and output a single vector for each masked patch.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Masked Siamese Networks for Label-Efficient Learning

Assran¹,

Caron²,

Misra³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available at https://github.com/facebookresearch/msn.

show abstract

Section: Related Workmentioning

confidence: 99%

“…loss, iBOT(Zhou et al, 2021) and SplitMask(El-Nouby et al, 2021) apply a joint-embedding loss to an output representing the global sequence (either the [CLS] token or a global average pool of the patch vectors). SplitMask shows that by using a patch-level loss, you can reduce the amount of unlabeled pre-training data.…”

mentioning

confidence: 99%

Masked Siamese Networks for Label-Efficient Learning

Assran¹,

Caron²,

Misra³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The method of DABS [63] also uses the idea of patch misplacement, but it does not have a way to handle degenerate learning and it does not show performance improvements. A technique that has proven very beneficial to improve the training efficiency of vision transformers is token dropping [1,36,25,10]. We extend this technique by randomizing the token dropping amount and including the case of no dropping to narrow the domain gap between pre-training and transfer.…”

Section: Related Workmentioning

confidence: 99%

DILEMMA: Self-Supervised Shape and Texture Learning with Transformers

Sameni¹,

Jenni²,

Favaro³

2022

Preprint

View full text Add to dashboard Cite

There is a growing belief that deep neural networks with a shape bias may exhibit better generalization capabilities than models with a texture bias, because shape is a more reliable indicator of the object category. However, we show experimentally that existing measures of shape bias are not stable predictors of generalization and argue that shape discrimination should not come at the expense of texture discrimination. Thus, we propose a pseudo-task to explicitly boost both shape and texture discriminability in models trained via self-supervised learning. For this purpose, we train a ViT to detect which input token has been combined with an incorrect positional embedding. To retain texture discrimination, the ViT is also trained as in MoCo with a student-teacher architecture and a contrastive loss over an extra learnable class token. We call our method DILEMMA, which stands for Detection of Incorrect Location EMbeddings with MAsked inputs. We evaluate our method through fine-tuning on several datasets and show that it outperforms MoCoV3 and DINO. Moreover, we show that when downstream tasks are strongly reliant on shape (such as in the YOGA-82 pose dataset), our pre-trained features yield a significant gain over prior work. Code will be released upon publication.Preprint. Under review.

show abstract

“…(MIM; Bao et al, 2021), which randomly masks out some input tokens and then recovers the masked content by conditioning on the visible context, is able to learn rich visual representations and shows promising performance on various vision benchmarks (Zhou et al, 2021;He et al, 2021;Xie et al, 2021;Dong et al, 2021;Wei et al, 2021;El-Nouby et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Corrupted Image Modeling for Self-Supervised Visual Pre-Training

Fang¹,

Liu²,

Bao³

et al. 2022

Preprint

View full text Add to dashboard Cite

We introduce Corrupted Image Modeling (CIM) for self-supervised visual pre-training. CIM uses an auxiliary generator with a small trainable BEiT to corrupt the input image instead of using artificial mask tokens, where some patches are randomly selected and replaced with plausible alternatives sampled from the BEiT output distribution. Given this corrupted image, an enhancer network learns to either recover all the original image pixels, or predict whether each visual token is replaced by a generator sample or not. The generator and the enhancer are simultaneously trained and synergistically updated. After pre-training, the enhancer can be used as a high-capacity visual encoder for downstream tasks. CIM is a general and flexible visual pre-training framework that is suitable for various network architectures. For the first time, CIM demonstrates that both ViT and CNN can learn rich visual representations using a unified, non-Siamese framework. Experimental results show that our approach achieves compelling results in vision benchmarks, such as ImageNet classification and ADE20K semantic segmentation. For example, 300-epoch CIM pretrained vanilla ViT-Base/16 and ResNet-50 obtain 83.3 and 80.6 Top-1 fine-tuning accuracy on ImageNet-1K image classification respectively. * Contribution during internship at Microsoft.

show abstract

Are Large-scale Datasets Necessary for Self-Supervised Pre-training?

Cited by 28 publications

References 0 publications

Masked Siamese Networks for Label-Efficient Learning

Masked Siamese Networks for Label-Efficient Learning

DILEMMA: Self-Supervised Shape and Texture Learning with Transformers

Corrupted Image Modeling for Self-Supervised Visual Pre-Training

Contact Info

Product

Resources

About