ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Xu, Yufei; Zhang, Qiming; Zhang, Jing; Tao, Dacheng

doi:10.48550/arxiv.2106.03348

Cited by 15 publications

(17 citation statements)

References 89 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A preliminary version of this work was presented in [85]. This paper extends the previous study by introducing three major improvements.…”

Section: Comparison To the Conference Versionmentioning

confidence: 67%

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Zhang¹,

Xu²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Vision transformers have shown great potential in various computer vision tasks owing to their strong capability to model long-range dependency using the self-attention mechanism. Nevertheless, they treat an image as a 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance, which is instead learned implicitly from large-scale training data with longer training schedules. In this paper, we propose a Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and can learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. The proposed two kinds of cells are stacked in both isotropic and multi-stage manners to formulate two families of ViTAE models, i.e., the vanilla ViTAE and ViTAEv2. Experiments on the ImageNet dataset as well as downstream tasks on the MS COCO, ADE20K, and AP10K datasets validate the superiority of our models over the baseline transformer models and concurrent

show abstract

“…A preliminary version of this work was presented in [85]. This paper extends the previous study by introducing three major improvements.…”

Section: Comparison To the Conference Versionmentioning

confidence: 67%

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Zhang¹,

Xu²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Therefore, current research mainly focuses on applying explicit inductive bias to vision transformers [5,18,33,37] and reducing the computational complexity from quadratic to linear [9,13]. XCiT [9] is one study that focused on reducing the computational complexity.…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, the non-linear embedding method (SNE) also improves the classification rates on the ImageNet dataset, compared to the baseline model which uses the linear embedding method. ViT-Ti [8] 5.7M T 68.7 T2T-ViT-7 [37] 4.3M T 71.7 DeiT-Ti [29] 5.7M T 72.2 Mobile-Former-96M [3] 4.6M C+T 72.8 LocalViT-T2T [18] 4.3M C+T 72.5 PiT-Ti [11] 4.9M C+T 73.0 ConViT-Ti [5] 5.7M C+T 73.1 ConT-Ti [34] 5.8M C+T 74.9 LocalViT-T [18] 5.9M C+T 74.8 ViTAE-T [33] 4.8M C+T 75.3 EfficientNet-B0 [28] 5.3M C 76.3 CeiT-T [36] 6.4M C+T 76.4 T2T-ViT-12 [37] 6.9M T 76.5 ConT-S [34] 10.1M C+T 76.5 CoaT-Lite Tiny [32] 5.7M C+T 76.6 Swin-1G [3,19] 7.3M T 77.3 XCiT-T12 (baseline) [9] 6.7M T 77.…”

Section: Imagenet Classificationmentioning

confidence: 99%

Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints

Ahn¹,

Hong²,

Ju³

et al. 2021

Preprint

View full text Add to dashboard Cite

A vision transformer (ViT) is the dominant model in the computer vision field. Despite numerous studies that mainly focus on dealing with inductive bias and complexity, there remains the problem of finding better transformer networks. For example, conventional transformer-based models usually use a projection layer for each query (Q), key (K), and value (V) embedding before multi-head self-attention. Insufficient consideration of semantic Q, K, and V embedding may lead to a performance drop. In this paper, we propose three types of structures for Q, K, and V embedding. The first structure utilizes two layers with ReLU, which is a non-linear embedding for Q, K, and V . The second involves sharing one of the non-linear layers to share knowledge among Q, K, and V . The third proposed structure shares all non-linear layers with code parameters. The codes are trainable, and the values determine the embedding process to be performed among Q, K, and V . Hence, we demonstrate the superior image classification performance of the proposed approaches in experiments compared to several state-of-the-art approaches. The proposed method achieved 71.4% with a few parameters (of 3.1M ) on the ImageNet-1k dataset compared to that required by the original transformer model of . Additionally, the method achieved 93.3% with only 2.9M parameters in transfer learning on average for the CIFAR-10, CIFAR-100, Stanford Cars datasets, and STL-10 datasets, which is better than the accuracy of 92.2% obtained via the original XCiT-N12 model.

show abstract

“…A large body of research has been devoted into improving efficiency and effectiveness of vision transformers (Dosovitskiy et al, 2020). Recent advances improve original vision transformers from various perspectives, such as data-efficient training (Touvron et al, 2020), adopting pyramid architectures (Wang et al, 2021a;Liu et al, 2021a;Heo et al, 2021), incorporating convolutional modules into the transformer Xu et al, 2021;d'Ascoli et al, 2021) and reducing computational costs by restricting the scope of the self-attention (Liu et al, 2021a;Dong et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Ripple Attention for Visual Perception with Sub-quadratic Complexity

Lin¹,

Pan²,

Kong³

2021

Preprint

View full text Add to dashboard Cite

Transformer architectures are now central to modeling in natural language processing tasks. At its heart is the attention mechanism, which enables effective modeling of long-term dependencies in a sequence. Recently, transformers have been successfully applied in the computer vision domain, where 2D images are first segmented into patches and then treated as 1D sequences. Such linearization, however, impairs the notion of spatial locality in images, which bears important visual clues. To bridge the gap, we propose ripple attention, a sub-quadratic attention mechanism for visual perception. In ripple attention, contributions of different tokens to a query are weighted with respect to their relative spatial distances in the 2D space. To favor correlations with vicinal tokens yet permit long-term dependencies, we derive the spatial weights through a stick-breaking transformation. We further design a dynamic programming algorithm that computes weighted contributions for all queries in linear observed time, taking advantage of the summed-area table and recent advances in linearized attention. Extensive experiments and analyses demonstrate the effectiveness of ripple attention on various visual tasks.

show abstract

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Cited by 15 publications

References 89 publications

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints

Ripple Attention for Visual Perception with Sub-quadratic Complexity

Contact Info

Product

Resources

About