A simple, efficient and scalable contrastive masked autoencoder for learning visual representations

Abstract

We introduce CAN, a simple, efficient and scalable method for self-supervised learning of visual representations. Our framework is a minimal and conceptually clean synthesis of (C) contrastive learning, (A) masked autoencoders, and (N) the noise prediction approach used in diffusion models. The learning mechanisms are complementary to one another: contrastive learning shapes the embedding space across a batch of image samples; masked autoencoders focus on reconstruction of the low-frequency spatial correlations in a single image sample; and noise prediction encourages the reconstruction of the high-frequency components of an image. The combined approach results in a robust, scalable and simple-to-implement algorithm. The training process is symmetric, with 50% of patches in both views being masked at random, yielding a considerable efficiency improvement over prior contrastive learning methods. Extensive empirical studies demonstrate that CAN achieves strong downstream performance under both linear and finetuning evaluations on transfer learning and robustness tasks. CAN outperforms MAE and SimCLR when pre-training on ImageNet, but is especially useful for pre-training on larger uncurated datasets such as JFT-300M: for linear probe on ImageNet, CAN achieves 75.4% compared to 73.4% for SimCLR and 64.1% for MAE. The finetuned performance on ImageNet of our ViT-L model is 86.1%, compared to 85.5% for SimCLR, and 85.4% for MAE. The overall FLOPs load of SimCLR is 70% higher than CAN for ViT-L models 1 . * Equal contribution, order chosen randomly. Work done during an internship at Google. 1 Code will be released.

MOMA:Distill from Self-Supervised Teachers

“…CAN (Mishra et al, 2022) applied mask on both branches in siamese network and optimized an InfoNCE loss (Oord et al, 2018), a reconstruction loss, and a denoising loss. CMAE (Huang et al, 2022) computed a reconstruction loss and a contrastive loss based on the decoder's outputs between the online branch and the target branch.…”

“…It presents competitive results against different benchmarks in computer vision. We hope our method provides an insight on transferring and adapting the knowledge from large-scale pre-trained models in a computationally efficient way.Recent studies (Chung et al, 2021) (Mishra et al, 2022 attempt to combine the power of contrastive learning and masked modelling, yielding promising results. They suggest that both paradigms are complementary with each other and can deliver stronger representations when they are combined into a unified framework.…”

“…Recent studies (Chung et al, 2021) (Mishra et al, 2022 attempt to combine the power of contrastive learning and masked modelling, yielding promising results. They suggest that both paradigms are complementary with each other and can deliver stronger representations when they are combined into a unified framework.…”

Regularized Contrastive Masked Autoencoder Model for Machinery Anomaly Detection Using Diffusion-Based Data Augmentation

“…Furthermore, MAE methods treat each sample independently in the loss function, whereas Contrastive methods explicitly consider the relationships between all samples in a batch by adjusting embedding distances. Given these differences, we posit that these two approaches are complementary [42], which leads to extracting different discriminative features from a given input. On the other hand, the generalization capability of Contrastive learning is influenced by four crucial factors [13].…”