CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Chen, Chun-Fu; Fan, Quanfu; Panda, Rameswar

doi:10.1109/iccv48922.2021.00041

Cited by 790 publications

(244 citation statements)

References 30 publications

Supporting

Mentioning

244

Contrasting

Order By: Relevance

“…non-overlapped). However, as pointed out by prior works like T2T [51], CrossViT [5], and PiT [19], using a stronger but still simple subnetwork for the tokenization could further improve the performance, especially for the smaller models. Thus, we adopt one convolutional layer with overlapped kernel and identical stride when generating local tokens, e.g.…”

Section: Rsa Regional Tokensmentioning

confidence: 99%

RegionViT: Regional-to-Local Attention for Vision Transformers

Chen¹,

Panda²,

Fan³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Vision transformer (ViT) has recently showed its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ a novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional selfattention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information. Extensive experiments on three vision tasks, including image classification, object detection and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models will be publicly available.Preprint. Under review.

show abstract

Section: Rsa Regional Tokensmentioning

confidence: 99%

RegionViT: Regional-to-Local Attention for Vision Transformers

Chen¹,

Panda²,

Fan³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Sliding window [44] 81.4 ----Shifted window [39] 81.3 42.2 39.1 41.5 Spatially Sep [12] 81.5 42.7 39.5 42.9 Sequential Axial [26] 81.5 40. 4 caused by other factors. The results are reported in Table 8a.…”

Section: Ablation Studymentioning

confidence: 99%

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Dong¹,

Bao²,

Chen³

et al. 2021

Preprint

114

View full text Add to dashboard Cite

We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token. To address this issue, we develop the Cross-Shaped Window self-attention mechanism for computing self-attention in the horizontal and vertical stripes in parallel that form a cross-shaped window, with each stripe obtained by splitting the input feature into stripes of equal width. We provide a detailed mathematical analysis of the effect of the stripe width and vary the stripe width for different layers of the Transformer network which achieves strong modeling capability while limiting the computation cost. We also introduce Locally-enhanced Positional Encoding (LePE), which handles the local positional information better than existing encoding schemes. LePE naturally supports arbitrary input resolutions, and is thus especially effective and friendly for downstream tasks. Incorporated with these designs and a hierarchical structure, CSWin Transformer demonstrates competitive performance on common vision tasks. Specifically, it achieves 85.4% Top-1 accuracy on ImageNet-1K without any extra training data or label, 53.9 box AP and 46.4 mask AP on the COCO detection task, and 51.7 mIOU on the ADE20K semantic segmentation task, surpassing previous state-of-the-art Swin Transformer backbone by +1.2, +2.0, +1.4, and +2.0 respectively under the similar FLOPs setting. By further pretraining on the larger dataset ImageNet-21K, we achieve 87.5% Top-1 accuracy on ImageNet-1K and state-of-the-art segmentation performance on ADE20K with 55.7 mIoU. The code and models will be available at https: //github.com/microsoft/CSWin-Transformer

show abstract

“…Comparison to Concurrent work. CrossViT [6] also utilizes different patch sizes (e.g., small and large) and dual-paths in a single-stage structure as ViT [16] and XCiT [17]. However, CrossViT's interactions between branches only occur through [CLS] tokens, while MPViT allows all patches of different scales to interact.…”

Section: Related Workmentioning

confidence: 99%

MPViT: Multi-Path Vision Transformer for Dense Prediction

Lee¹,

Lee²,

Willette³

et al. 2021

Preprint

View full text Add to dashboard Cite

Dense computer vision tasks such as object detection and segmentation require effective multi-scale feature representation for detecting or classifying objects or regions with varying sizes. While Convolutional Neural Networks (CNNs) have been the dominant architectures for such tasks, recently introduced Vision Transformers (ViTs) aim to replace them as a backbone. Similar to CNNs, ViTs build a simple multi-stage structure (i.e., fine-to-coarse) for multi-scale representation with single-scale patches. In this work, with a different perspective from existing Transformers, we explore multi-scale patch embedding and multi-path structure, constructing the Multi-Path Vision Transformer (MPViT). MPViT embeds features of the same size (i.e., sequence length) with patches of different scales simultaneously by using overlapping convolutional patch embedding. Tokens of different scales are then independently fed into the Transformer encoders via multiple paths and the resulting features are aggregated, enabling both fine and coarse feature representations at the same feature level. Thanks to the diverse, multi-scale feature representations, our MPViTs scaling from tiny (5M) to base (73M) consistently achieve superior performance over state-of-theart Vision Transformers on ImageNet classification, object detection, instance segmentation, and semantic segmentation. These extensive results demonstrate that MPViT can serve as a versatile backbone network for various vision tasks. Code will be made publicly available at https: //git.io/MPViT.

show abstract

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Cited by 790 publications

References 30 publications

RegionViT: Regional-to-Local Attention for Vision Transformers

RegionViT: Regional-to-Local Attention for Vision Transformers

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

MPViT: Multi-Path Vision Transformer for Dense Prediction

Contact Info

Product

Resources

About