Revealing the Dark Secrets of Masked Image Modeling

Abstract

Masked image modeling (MIM) as pre-training is shown to be effective for numerous vision downstream tasks, but how and where MIM works remain unclear. In this paper, we compare MIM with the long-dominant supervised pre-trained models from two perspectives, the visualizations and the experiments, to uncover their key representational differences. From the visualizations, we find that MIM brings locality inductive bias to all layers of the trained models, but supervised models tend to focus locally at lower layers but more globally at higher layers. That may be the reason why MIM helps Vision Transformers that have a very large receptive field to optimize. Using MIM, the model can maintain a large diversity on attention heads in all layers. But for supervised models, the diversity on attention heads almost disappears from the last three layers and less diversity harms the fine-tuning performance. From the experiments, we find that MIM models can perform significantly better on geometric and motion tasks with weak semantics or fine-grained classification tasks, than their supervised counterparts. Without bells and whistles, a standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the categories are sufficiently covered by the supervised pre-training, MIM models can still achieve highly competitive transfer performance. With a deeper understanding of MIM, we hope that our work can inspire new and solid research in this direction. * Equal Contribution. The work is done when Zhenda Xie, Zigang Geng, and Jingcheng Hu are long-term interns at Microsoft Research Asia. † Contact person.

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

“…Hence, VoxFormer with stereo depth performs best. Note that our framework can be integrated with any state-of-theart depth models, so using a stronger existing depth predictor [74][75][76] could enhance our SSC performance. Meanwhile, VoxFormer can be further promoted along with the advancement of depth estimation.…”

GFM: Building Geospatial Foundation Models via Continual Pretraining

“…In this work, we develop our pretraining objective based on a masked image modeling approach like [20,44]. MIM has recently been shown to be particularly effective in the natural image domain, surpassing many contrastive works and being shown to be friendlier to downstream optimization [3,20,43,44,49]. Exploration of the masked image modeling framework in geospatial applications is still in its early stages, and could help allivate some concerns with contrastive approaches in this domain.…”

SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling

“…Mixing-up (Wickstrøm et al, 2022) exploits a data augmentation scheme in which new samples are generated by mixing two data samples and the model is optimized to predict the mixing weights. Note that contrastive learning mainly focuses on the highlevel information (Xie et al, 2022a) and the series-wise or patch-wise representations inherently mismatch the lowlevel tasks, such as time series forecasting. Thus, in this paper, we focus on the masked modeling paradigm.…”

“…Masked modeling has been explored in stacked denoising autoencoders (Vincent et al, 2010), where the masking is viewed as adding noise to the original data and the masked modeling is to project the masked data from the neighborhood back to the original manifold, namely denoising. Recently, it has been widely used in pre-training, which can learn valuable low-level information from data unsupervisedly (Xie et al, 2022a). Inspired by the manifold perspective, we go beyond the classical denoising process and project the masked data back to the manifold by aggregating multiple masked time series within the neighborhood.…”