Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality

Li, Xiang; Wang, Wenhai; Yang, Liu; Yang, Jian

doi:10.48550/arxiv.2205.10063

Cited by 11 publications

(16 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately, hViT cannot be directly applied to enable MAE pretraining because the local window attention used in hViT makes it difficult to handle randomly masked patches as in MAE. Recently, Uniform Masking MAE (UM-MAE) [154] is proposed to empower MAE with hViTs, which introduces a two-stage pipeline: sampling and masking. It starts by randomly sampling a portion of patches (25% reported in the paper) from each block, and then follows by masking additional patches on top of the sampled ones.…”

Section: Learning By Reconstructionmentioning

confidence: 99%

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Zhou¹,

Li²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

The Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A pretrained foundation model, such as BERT, GPT-3, MAE, DALLE-E, and ChatGPT, is trained on large-scale data which provides a reasonable parameter initialization for a wide range of downstream applications. The idea of pretraining behind PFMs plays an important role in the application of large models. As a type of transfer learning paradigm, pretraining is applied in computer vision with frozen and fine-tuning techniques, showing promising performance. Word embedding in the natural language process can be also regarded as a type of pertaining, but it suffers many problems such as polysemy. Different from previous methods that apply convolution and recurrent modules for feature extractions, the generative pre-training (GPT) method applies Transformer as the feature extractor and is trained on large datasets with an autoregressive paradigm. Similarly, the BERT apples transformers to train on large datasets as a contextual language model. Recently, the ChatGPT shows promising success on large language models, which applies an autoregressive language model with zero shot or few show prompting. With the extraordinary success of PFMs, AI has made waves in a variety of fields over the past few years. Considerable methods, datasets, and evaluation metrics have been proposed in the literature, the need is raising for an updated survey.This study provides a comprehensive review of recent research advancements, current and future challenges, and opportunities for PFMs in text, image, graph, as well as other data modalities. We first review the basic components and existing pretraining in natural language processing, computer vision, and graph learning. We then discuss other advanced PFMs for other data modalities and unified PFMs considering the data quality and quantity. Besides, we discuss relevant research about the fundamentals of the PFM, including model efficiency and compression, security, and privacy. Finally, we lay out key implications, future research directions, challenges, and open problems. We hope this survey can shed light on the research of PFMs on the scalability, reasoning ability, cross-domain ability, user-friendly interactive ability, security and privacy-preserving ability for artificial general intelligence.

show abstract

Section: Learning By Reconstructionmentioning

confidence: 99%

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Zhou¹,

Li²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…An inevitable bottleneck for the industrial applications of MIM is that these models typically require huge computational resources and long pre-training duration. To this end, some works accelerate the encoding process via the asymmetric encoder-decoder strategy [24,31] or lessening the input patches [8,35]. Only accelerating the encoding process sometimes doesn't really speed up the representation learning, like GreenMIM vs. SimMIM 192 in Fig.…”

Section: Related Workmentioning

confidence: 99%

“…Further, GreenMIM [31] extends the asymmetric encoder-decoder strategy to hierarchical vision transformers (e.g., Swin [39]). Besides, [8,22,35] shrinks the input resolution to lessen the input patches, thereby reducing the computational burden. However, they all aim to accelerate the encoding process rather than the representation learning.…”

Section: Introductionmentioning

confidence: 99%

Masked Image Modeling with Local Multi-Scale Reconstruction

Wang¹,

Wang²,

Guo³

et al. 2023

Preprint

View full text Add to dashboard Cite

Masked Image Modeling (MIM) achieves outstanding success in self-supervised representation learning. Unfortunately, MIM models typically have huge computational burden and slow learning process, which is an inevitable obstacle for their industrial applications. Although the lower layers play the key role in MIM, existing MIM models conduct reconstruction task only at the top layer of encoder. The lower layers are not explicitly guided and the interaction among their patches is only used for calculating new activations. Considering the reconstruction task requires non-trivial inter-patch interactions to reason target signals, we apply it to multiple local layers including lower and upper layers. Further, since the multiple layers expect to learn the information of different scales, we design local multiscale reconstruction, where the lower and upper layers reconstruct fine-scale and coarse-scale supervision signals respectively. This design not only accelerates the representation learning process by explicitly guiding multiple layers, but also facilitates multi-scale semantical understanding to the input. Extensive experiments show that with significantly less pre-training burden, our model achieves comparable or better performance on classification, detection and segmentation tasks than existing MIM models. Code is available with both MindSpore and PyTorch.

show abstract

“…CAE [37] separates the encoder representation from the prediction task and makes predictions in the latent representation space from visible patches to mask patches. UM-MAE [38] successfully uses quadratic masking strategy to achieve self-supervision in pyramid networks like Swin Transformer [13], PVT [14], etc. ConvMAE [39] presents a simple self-supervised learning framework with a block-wise masking strategy, which demonstrates that multi-scale features from supervised encoders can improve the performance of downstream tasks.…”

Section: Related Workmentioning

confidence: 99%

“…The very recent approach Green-MAE [40] is similar to our approach, allowing the hierarchical models to discard masked patches and operate only on the visible ones. Our CoTMAE benefits from the development of hybrid convolutional-transformer pyramid networks and useful experience gained from recent works [34][35][36][37][38][39][40][41][42].…”

Section: Related Workmentioning

confidence: 99%

CoTMAE:Hybrid Convolution-Transformer Pyramid Network Meets Masked Autoencoder

Li¹,

Yang²,

Hu³

et al. 2022

Asian Society for Precision Engineering and Nanotechnology (ASPEN 2022)

View full text Add to dashboard Cite

Vision Transformer (ViT) has become the most popular architecture for existing vision tasks, but it is difficult to apply to the industrial domain due to its heavy computational cost of its self-attention mechanism. Masked AutoEncoder (MAE) has recently led the trend of self-supervised learning with a simple, scalable, and efficient ViT-based asymmetric encoder-decoder architecture. To mitigate the quadratic complexity of self-attention, we design a hybrid convolution-transformer pyramid network that effectively combines the respective advantages of convolution and self-attention. However, it is still unclear how our convolution-transformer pyramid network can be adopted in MAE pre-training, as it uses the local convolution operation, making it difficult to handle random sequences with only partial visual tokens. In this paper, we present a novel and efficient masked image modeling (MIM) approach, convolutional-contextual transformer masked autoencoder (CoTMAE). The pipeline of CoTMAE consists of: (i) a window masking (WM) strategy that ensures computational efficiency, (ii) an encoder that only takes visible patches as input to our hybrid convolution-transformer network, (iii) a multi-scale fusion module that enhances the output features of the encoder, which allows the decoder to focus on the reconstruction task. (iv) a feature alignment module that handles the distribution of encoded features and masked patches, and (v) a decoder that reconstructs the missing pixels of the masked patches. Specifically, WM directly divides the original image into equal-sized windows, using a random mask strategy within each window. Afterwards, only visible patches are reordered and reorganized into images as input to the hybrid convolution-transformer pyramid network. Our WM significantly improves the training efficiency of hybrid convolution-transformer networks and reduces GPU memory, while maintaining a competitive advantage with supervised training models in downstream tasks. We demonstrate that CoTMAE successfully enables self-supervised pre-training of a hybrid convolution-transformer pyramid network and achieves good fine-tuning performance on instance segmentation datasets. The encoder of CoTMAE is trained on ImageNet-1K dataset classification and fine-tuned on COCO 2017 dataset to achieve 52.9% APbox and 45.8% APmask. On industrial instance segmentation datasets, CoTMAE shows better fine-tuning performance than supervised models.

show abstract

Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality

Cited by 11 publications

References 33 publications

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

Masked Image Modeling with Local Multi-Scale Reconstruction

CoTMAE:Hybrid Convolution-Transformer Pyramid Network Meets Masked Autoencoder

Contact Info

Product

Resources

About