MixMIM: Mixed and Masked Image Modeling for Efficient Visual Representation Learning

Liu, Jihao; Huang, Xin; Liu, Yu; Li, Hongsheng

doi:10.48550/arxiv.2205.13137

Cited by 12 publications

(27 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SimMIM [45] Swin-B RGB 100% 60% 800 84.0 MCMAE [11] CViT-B RGB 25% 75% 1600 85.0 MixMIM [29] MixMIM-B RGB 100% 100% 600 85.1 CMAE [19] CViT-B RGB 25% 75% 1600 85.3…”

Section: Methodsmentioning

confidence: 99%

Stare at What You See: Masked Image Modeling without Reconstruction

Hou¹,

Gao²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Masked Autoencoders (MAE) have been prevailing paradigms for large-scale vision representation pretraining. By reconstructing masked image patches from a small portion of visible image regions, MAE forces the model to infer semantic correlation within an image. Recently, some approaches apply semantic-rich teacher models to extract image features as the reconstruction target, leading to better performance. However, unlike the lowlevel features such as pixel values, we argue the features extracted by powerful teacher models already encode rich semantic correlation across regions in an intact image. This raises one question: is reconstruction necessary in Masked Image Modeling (MIM) with a teacher model? In this paper, we propose an efficient MIM paradigm named MaskAlign. MaskAlign simply learns the consistency of visible patch features extracted by the student model and intact image features extracted by the teacher model. To further advance the performance and tackle the problem of input inconsistency between the student and teacher model, we propose a Dynamic Alignment (DA) module to apply learnable alignment. Our experimental results demonstrate that masked modeling does not lose effectiveness even without reconstruction on masked regions. Combined with Dynamic Alignment, MaskAlign can achieve state-of-theart performance with much higher efficiency. Code and models will be available at https://github.com/ OpenPerceptionX/maskalign.

show abstract

“…SimMIM [45] Swin-B RGB 100% 60% 800 84.0 MCMAE [11] CViT-B RGB 25% 75% 1600 85.0 MixMIM [29] MixMIM-B RGB 100% 100% 600 85.1 CMAE [19] CViT-B RGB 25% 75% 1600 85.3…”

Section: Methodsmentioning

confidence: 99%

Stare at What You See: Masked Image Modeling without Reconstruction

Hou¹,

Gao²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Specifically, during pretraining, we partition the training images into patches and feed a portion of them into the encoder following Masked Autoencoder [14]. Our GeoMIM decoder then uses these encoded visible to-kens to reconstruct the pretrained LiDAR model's BEV feature in the BEV space instead of commonly used RGB pixels [47,14,30] or depth points [3] as in existing MAE frameworks. To achieve this PV to BEV reconstruction, we first devise two branches to decouple the semantic and geometric parts, with one branch completing dense PV features and the other reconstructing the depth map.…”

Section: Pretrain Supervision Finetunementioning

confidence: 99%

Towards Better 3D Knowledge Transfer via Masked Image Modeling for Multi-view 3D Understanding

Liu¹,

Wang²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Multi-view camera-based 3D detection is a challenging problem in computer vision. Recent works leverage a pretrained LiDAR detection model to transfer knowledge to a camera-based student network. However, we argue that there is a major domain gap between the LiDAR BEV features and the camera-based BEV features, as they have different characteristics and are derived from different sources. In this paper, we propose Geometry Enhanced Masked Image Modeling (GeoMIM) to transfer the knowledge of the LiDAR model in a pretrain-finetune paradigm for improving the multi-view camera-based 3D detection. GeoMIM is a multicamera vision transformer with Cross-View Attention (CVA) blocks that uses LiDAR BEV features encoded by the pretrained BEV model as learning targets. During pretraining, GeoMIM's decoder has a semantic branch completing dense perspective-view features and the other geometry branch reconstructing dense perspective-view depth maps. The depth branch is designed to be camera-aware by inputting the camera's parameters for better transfer capability. Extensive results demonstrate that GeoMIM outperforms existing methods on nuScenes benchmark, achieving state-of-the-art performance for camera-based 3D object detection and 3D segmentation.

show abstract

“…In the past few years, contrastive learning is very popular, which aims to learn invariances from different augmented views of images [4,12,16]. Recently, Masked Image Modeling (MIM) [11,15,28,40] become more and more prevalent for vision transformers. MIM is the task that reconstructs image content from a masked image.…”

Section: Related Workmentioning

confidence: 99%

Mutual Exclusive Modulator for Long-Tailed Recognition

Long¹,

Zhang²,

Li³

2023

Preprint

View full text Add to dashboard Cite

The long-tailed recognition (LTR) is the task of learning high-performance classifiers given extremely imbalanced training samples between categories. Most of the existing works address the problem by either enhancing the features of tail classes or re-balancing the classifiers to reduce the inductive bias. In this paper, we try to look into the root cause of the LTR task, i.e., training samples for each class are greatly imbalanced, and propose a straightforward solution. We split the categories into three groups, i.e., many, medium and few, according to the number of training images. The three groups of categories are separately predicted to reduce the difficulty for classification. This idea naturally arises a new problem of how to assign a given sample to the right class groups? We introduce a mutual exclusive modulator which can estimate the probability of an image belonging to each group. Particularly, the modulator consists of a light-weight module and learned with a mutual exclusive objective. Hence, the output probabilities of the modulator encode the data volume clues of the training dataset. They are further utilized as prior information to guide the prediction of the classifier. We conduct extensive experiments on multiple datasets, e.g., ImageNet-LT, Place-LT and iNaturalist 2018 to evaluate the proposed approach. Our method achieves competitive performance compared to the state-of-the-art benchmarks.

show abstract

MixMIM: Mixed and Masked Image Modeling for Efficient Visual Representation Learning

Cited by 12 publications

References 27 publications

Stare at What You See: Masked Image Modeling without Reconstruction

Stare at What You See: Masked Image Modeling without Reconstruction

Towards Better 3D Knowledge Transfer via Masked Image Modeling for Multi-view 3D Understanding

Mutual Exclusive Modulator for Long-Tailed Recognition

Contact Info

Product

Resources

About