Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors

Abstract

Figure 1. We present Mask3D, which learns to embed 3D priors to 2D representations for image understanding tasks, based on a selfsupervised pre-training formulation from single RGB-D views, without requiring any camera pose or multi-view correspondence information. Our pre-training takes masked RGB and depth patches as input to reconstruct the dense depth map, and the pre-trained color backbone is used to fine-tune various downstream image understanding tasks. This results in effective ViT pre-training for a variety of downstream tasks and datasets.