Green Hierarchical Vision Transformer for Masked Image Modeling

Huang, Lang; You, Suya; Zheng, Mingkai; Wang, Fei; Chen, Qian; Yamasaki, Toshihiko

doi:10.48550/arxiv.2205.13515

Cited by 4 publications

(20 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ConvMAE [39] presents a simple self-supervised learning framework with a block-wise masking strategy, which demonstrates that multi-scale features from supervised encoders can improve the performance of downstream tasks. The very recent approach Green-MAE [40] is similar to our approach, allowing the hierarchical models to discard masked patches and operate only on the visible ones. Our CoTMAE benefits from the development of hybrid convolutional-transformer pyramid networks and useful experience gained from recent works [34][35][36][37][38][39][40][41][42].…”

Section: Related Workmentioning

confidence: 99%

“…The very recent approach Green-MAE [40] is similar to our approach, allowing the hierarchical models to discard masked patches and operate only on the visible ones. Our CoTMAE benefits from the development of hybrid convolutional-transformer pyramid networks and useful experience gained from recent works [34][35][36][37][38][39][40][41][42].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

CoTMAE:Hybrid Convolution-Transformer Pyramid Network Meets Masked Autoencoder

Li¹,

Yang²,

Hu³

et al. 2022

Asian Society for Precision Engineering and Nanotechnology (ASPEN 2022)

View full text Add to dashboard Cite

Vision Transformer (ViT) has become the most popular architecture for existing vision tasks, but it is difficult to apply to the industrial domain due to its heavy computational cost of its self-attention mechanism. Masked AutoEncoder (MAE) has recently led the trend of self-supervised learning with a simple, scalable, and efficient ViT-based asymmetric encoder-decoder architecture. To mitigate the quadratic complexity of self-attention, we design a hybrid convolution-transformer pyramid network that effectively combines the respective advantages of convolution and self-attention. However, it is still unclear how our convolution-transformer pyramid network can be adopted in MAE pre-training, as it uses the local convolution operation, making it difficult to handle random sequences with only partial visual tokens. In this paper, we present a novel and efficient masked image modeling (MIM) approach, convolutional-contextual transformer masked autoencoder (CoTMAE). The pipeline of CoTMAE consists of: (i) a window masking (WM) strategy that ensures computational efficiency, (ii) an encoder that only takes visible patches as input to our hybrid convolution-transformer network, (iii) a multi-scale fusion module that enhances the output features of the encoder, which allows the decoder to focus on the reconstruction task. (iv) a feature alignment module that handles the distribution of encoded features and masked patches, and (v) a decoder that reconstructs the missing pixels of the masked patches. Specifically, WM directly divides the original image into equal-sized windows, using a random mask strategy within each window. Afterwards, only visible patches are reordered and reorganized into images as input to the hybrid convolution-transformer pyramid network. Our WM significantly improves the training efficiency of hybrid convolution-transformer networks and reduces GPU memory, while maintaining a competitive advantage with supervised training models in downstream tasks. We demonstrate that CoTMAE successfully enables self-supervised pre-training of a hybrid convolution-transformer pyramid network and achieves good fine-tuning performance on instance segmentation datasets. The encoder of CoTMAE is trained on ImageNet-1K dataset classification and fine-tuned on COCO 2017 dataset to achieve 52.9% APbox and 45.8% APmask. On industrial instance segmentation datasets, CoTMAE shows better fine-tuning performance than supervised models.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

CoTMAE:Hybrid Convolution-Transformer Pyramid Network Meets Masked Autoencoder

Li¹,

Yang²,

Hu³

et al. 2022

Asian Society for Precision Engineering and Nanotechnology (ASPEN 2022)

View full text Add to dashboard Cite

show abstract

“…Despite superior performance on various downstream tasks, these models have huge computational burden and slow learning process [31]. They typically require thousands of GPU Hours for pre-training on ImageNet-1K to get generalizing representations.…”

Section: Introductionmentioning

confidence: 99%

“…To this end, MAE [24] pioneered the asymmetric encoder-decoder strategy, where the costly encoder only operates few visible patches and the lightweight decoder takes all the patches as input for prediction. Further, GreenMIM [31] extends the asymmetric encoder-decoder strategy to hierarchical vision transformers (e.g., Swin [39]). Besides, [8,22,35] shrinks the input resolution to lessen the input patches, thereby reducing the computational burden.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Masked Image Modeling with Local Multi-Scale Reconstruction

Wang¹,

Wang²,

Guo³

et al. 2023

Preprint

View full text Add to dashboard Cite

Masked Image Modeling (MIM) achieves outstanding success in self-supervised representation learning. Unfortunately, MIM models typically have huge computational burden and slow learning process, which is an inevitable obstacle for their industrial applications. Although the lower layers play the key role in MIM, existing MIM models conduct reconstruction task only at the top layer of encoder. The lower layers are not explicitly guided and the interaction among their patches is only used for calculating new activations. Considering the reconstruction task requires non-trivial inter-patch interactions to reason target signals, we apply it to multiple local layers including lower and upper layers. Further, since the multiple layers expect to learn the information of different scales, we design local multiscale reconstruction, where the lower and upper layers reconstruct fine-scale and coarse-scale supervision signals respectively. This design not only accelerates the representation learning process by explicitly guiding multiple layers, but also facilitates multi-scale semantical understanding to the input. Extensive experiments show that with significantly less pre-training burden, our model achieves comparable or better performance on classification, detection and segmentation tasks than existing MIM models. Code is available with both MindSpore and PyTorch.

show abstract

Masked Generative Light Field Prompting for Pixel-Level Structure Segmentations

Wang,

Shi,

Cheng

et al. 2024

Research

View full text Add to dashboard Cite

Pixel-level structure segmentations have attracted considerable attention, playing a crucial role in autonomous driving within the metaverse and enhancing comprehension in light field-based machine vision. However, current light field modeling methods fail to integrate appearance and geometric structural information into a coherent semantic space, thereby limiting the capability of light field transmission for visual knowledge. In this paper, we propose a general light field modeling method for pixel-level structure segmentation, comprising a generative light field prompting encoder (LF-GPE) and a prompt-based masked light field pretraining (LF-PMP) network. Our LF-GPE, serving as a light field backbone, can extract both appearance and geometric structural cues simultaneously. It aligns these features into a unified visual space, facilitating semantic interaction. Meanwhile, our LF-PMP, during the pretraining phase, integrates a mixed light field and a multi-view light field reconstruction. It prioritizes considering the geometric structural properties of the light field, enabling the light field backbone to accumulate a wealth of prior knowledge. We evaluate our pretrained LF-GPE on two downstream tasks: light field salient object detection and semantic segmentation. Experimental results demonstrate that LF-GPE can effectively learn high-quality light field features and achieve highly competitive performance in pixel-level segmentation tasks.

show abstract

Green Hierarchical Vision Transformer for Masked Image Modeling

Cited by 4 publications

References 50 publications

CoTMAE:Hybrid Convolution-Transformer Pyramid Network Meets Masked Autoencoder

CoTMAE:Hybrid Convolution-Transformer Pyramid Network Meets Masked Autoencoder

Masked Image Modeling with Local Multi-Scale Reconstruction

Masked Generative Light Field Prompting for Pixel-Level Structure Segmentations

Contact Info

Product

Resources

About