ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Zhang, Qiming; Xu, Yan; Zhang, Jing; Tao, Dacheng

doi:10.48550/arxiv.2202.10108

Cited by 15 publications

(21 citation statements)

References 68 publications

(130 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, various configurations of an MLP-like variant offer limited gains despite the increased number of parameters. Recently, the visual community has conducted some scaling-up research on the vision Transformers with self-supervised pre-training, including V-MoE, 90 Swinv2, 117 and ViTAEv2, 140 which afford a considerable performance boost. Nevertheless, scaling-up techniques specific to MLPs need further exploration.…”

Section: Discussionmentioning

confidence: 99%

Are we ready for a new paradigm shift? A survey on visual deep MLP

Liu

Tao

et al. 2022

Patterns

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

Are we ready for a new paradigm shift? A survey on visual deep MLP

Liu

Tao

et al. 2022

Patterns

View full text Add to dashboard Cite

“…We also use LayerScale [70] to train deep models. Like previous studies [5,66], we further fine tune iFormer on the input size of 384 × 384, with the weight decay of 1 × 10 −8 , learning rate of 1 × 10 −5 , batch size of 512. For fairness, we adopt Timm [71] to implement and train iFormer.…”

Section: Results On Image Classificationmentioning

confidence: 99%

Inception Transformer

Si¹,

Yu²,

Zhou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recent studies show that Transformer has strong capability of building long-range dependencies, yet is incompetent in capturing high frequencies that predominantly convey local information. To tackle this issue, we present a novel and general-purpose Inception Transformer, or iFormer for short, that effectively learns comprehensive features with both high-and low-frequency information in visual data. Specifically, we design an Inception mixer to explicitly graft the advantages of convolution and max-pooling for capturing the high-frequency information to Transformers. Different from recent hybrid frameworks, the Inception mixer brings greater efficiency through a channel splitting mechanism to adopt parallel convolution/max-pooling path and self-attention path as high-and low-frequency mixers, while having the flexibility to model discriminative information scattered within a wide frequency range. Considering that bottom layers play more roles in capturing high-frequency details while top layers more in modeling low-frequency global information, we further introduce a frequency ramp structure, i.e., gradually decreasing the dimensions fed to the high-frequency mixer and increasing those to the low-frequency mixer, which can effectively trade-off high-and lowfrequency components across different layers. We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation. For example, our iFormer-S hits the top-1 accuracy of 83.4% on ImageNet-1K, much higher than DeiT-S by 3.6%, and even slightly better than much bigger model Swin-B (83.3%) with only 1/4 parameters and 1/3 FLOPs. Code and models will be released at https://github.com/sail-sg/iFormer. * Equal contribution. Weihao Yu did this work during an internship at Sea AI Lab.Preprint. Under review.

show abstract

“…We finetune our model end-to-end on MS COCO [34] for the object detection and instance segmentation tasks. We replace the ViT backbone with our pretrained LoMaR model in the ViTDet [33] and ViTAE [60] frameworks. We report object detection results in AP box and instance segmentation results in AP mask .…”

Section: Comparison With Other Self-supervised Approachesmentioning

confidence: 99%

Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction

Hu¹,

Li²,

Elhoseiny³

2022

Preprint

View full text Add to dashboard Cite

Self-supervised learning for computer vision has achieved tremendous progress and improved many downstream vision tasks such as image classification, semantic segmentation, and object detection. Among these, generative self-supervised vision learning approaches such as MAE and BEiT show promising performance. However, their global masked reconstruction mechanism is computationally demanding. To address this issue, we propose local masked reconstruction (LoMaR), a simple yet effective approach that performs masked reconstruction within a small window of 7×7 patches on a simple Transformer encoder, improving the trade-off between efficiency and accuracy compared to global masked reconstruction over the entire image. Extensive experiments show that LoMaR reaches 84.1% top-1 accuracy on ImageNet-1K classification, outperforming MAE by 0.5%. After finetuning the pretrained LoMaR on 384×384 images, it can reach 85.4% top-1 accuracy, surpassing MAE by 0.6%. On MS COCO, LoMaR outperforms MAE by 0.5 AP box on object detection and 0.5 AP mask on instance segmentation. LoMaR is especially more computation-efficient on pretraining high-resolution images, e.g., it is 3.1× faster than MAE with 0.2% higher classification accuracy on pretraining 448×448 images. This local masked reconstruction learning mechanism can be easily integrated into any other generative self-supervised learning approach. Our code is publicly available in https://github.com/junchen14/LoMaR.Preprint. Under review.

show abstract

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Cited by 15 publications

References 68 publications

Are we ready for a new paradigm shift? A survey on visual deep MLP

Are we ready for a new paradigm shift? A survey on visual deep MLP

Inception Transformer

Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction

Contact Info

Product

Resources

About