CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Dong, Xiaoyi; Bao, Jianmin; Chen, Dongdong; Zhang, Weiming; Yu, Nenghai; Liu, Yuan; Chen, Dong; Guo, Baining

doi:10.48550/arxiv.2107.00652

Cited by 75 publications

(123 citation statements)

References 68 publications

Supporting

Mentioning

123

Contrasting

Order By: Relevance

“…Transformer and Focal Transformer. For more fair comparison, we adopt the same positional encoding LePE (Dong et al, 2021) to PVTv2, Swin and Focal transformer. As shown in Table 8, QuadTree attention obtain consistently better performance than Swin and PVTv2 in both classification task and detection task.…”

Section: Discussionmentioning

confidence: 99%

“…ImageNet-1K COCO (RetinaNet) Flops (G) Top-1 (%) Mem. (MB) AP AP50 AP75 PVTv2 0.6 70.5 574 37.2 57.2 39.5 PVTv2+LePE (Dong et al, 2021) 0.6 70.9 574 37.6 57.8 39.9 Swin 0…”

Section: Discussionmentioning

confidence: 99%

“…The computation of attention is permutation invariant to tokens, and thus positional information is missed. To address this problem, we adopt the locally-enhanced positional encoding (LePE) (Dong et al, 2021) at each level to design a multiscale position encoding. Specifically, for level l, we apply unshared depth-wise convolution layers to value tokens V l to encode the positional information.…”

Section: Quadtree Attentionmentioning

confidence: 99%

See 2 more Smart Citations

QuadTree Attention for Vision Transformers

Tang¹,

Zhang²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformers have been successful in many vision tasks, thanks to their capability of capturing long-range dependency. However, their quadratic computational complexity poses a major obstacle for applying them to vision tasks requiring dense predictions, such as object detection, feature matching, stereo, etc. We introduce QuadTree Attention, which reduces the computational complexity from quadratic to linear. Our quadtree transformer builds token pyramids and computes attention in a coarse-to-fine manner. At each level, the top K patches with the highest attention scores are selected, such that at the next level, attention is only evaluated within the relevant regions corresponding to these top K patches. We demonstrate that quadtree attention achieves state-of-theart performance in various vision tasks, e.g. with 4.0% improvement in feature matching on ScanNet, about 50% flops reduction in stereo matching, 0.4-1.5% improvement in top-1 accuracy on ImageNet classification, 1.2-1.8% improvement on COCO object detection, and 0.7-2.4% improvement on semantic segmentation over previous state-of-the-art transformers. The codes are available at https://github.com/Tangshitao/QuadtreeAttention.

show abstract

Section: Discussionmentioning

confidence: 99%

“…ImageNet-1K COCO (RetinaNet) Flops (G) Top-1 (%) Mem. (MB) AP AP50 AP75 PVTv2 0.6 70.5 574 37.2 57.2 39.5 PVTv2+LePE (Dong et al, 2021) 0.6 70.9 574 37.6 57.8 39.9 Swin 0…”

Section: Discussionmentioning

confidence: 99%

Section: Quadtree Attentionmentioning

confidence: 99%

See 1 more Smart Citation

QuadTree Attention for Vision Transformers

Tang¹,

Zhang²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Vision transformers [2,18,19,22,31,49,70,71,88] treat an image as a set of patches and model their interactions with transformer-based architectures [74]. Recent works adding vision priors such as multi-scale feature hierarchies [22,31,49,80,88] or local structure modeling [9,18,49] have shown to be effective. They have also been generalized from the image to video domain [3,22,51,54].…”

Section: Related Workmentioning

confidence: 99%

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition

Wu¹,

Li²,

Mangalam³

et al. 2022

Preprint

View full text Add to dashboard Cite

While today's video recognition systems parse snapshots or short clips accurately, they cannot connect the dots and reason across a longer range of time yet. Most existing video architectures can only process <5 seconds of a video without hitting the computation or memory bottlenecks.In this paper, we propose a new strategy to overcome this challenge. Instead of trying to process more frames at once like most existing methods, we propose to process videos in an online fashion and cache "memory" at each iteration. Through the memory, the model can reference prior context for long-term modeling, with only a marginal cost. Based on this idea, we build MeMViT, a Memoryaugmented Multiscale Vision Transformer, that has a temporal support 30×longer than existing models with only 4.5% more compute; traditional methods need >3,000% more compute to do the same. On a wide range of settings, the increased temporal support enabled by MeMViT brings large gains in recognition accuracy consistently. MeMViT obtains state-of-the-art results on the AVA, EPIC-Kitchens-100 action classification, and action anticipation datasets. Code and models will be made publicly available.

show abstract

“…Recently, the pioneering work ViT [22] successfully applies the pure transformer-based architecture to computer vision, revealing the potential of transformer in handling visual tasks. Lots of follow-up studies are proposed [4,5,9,12,18,21,23,24,[27][28][29]31,38,41,43,45,50,52,56,76,77,80,81,84]. Many of them analyze the ViT [15,17,26,32,44,55,69,73,75,82] and improve it via introducing locality to earlier layers [11,17,48,64,79,83,87].…”

Section: Related Workmentioning

confidence: 99%

ELSA: Enhanced Local Self-Attention for Vision Transformer

Zhou¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-attention is powerful in modeling long-range dependencies, but it is weak in local finer-level feature learning. The performance of local self-attention (LSA) is just on par with convolution and inferior to dynamic filters, which puzzles researchers on whether to use LSA or its counterparts, which one is better, and what makes LSA mediocre. To clarify these, we comprehensively investigate LSA and its counterparts from two sides: channel setting and spatial processing. We find that the devil lies in the generation and application of spatial attention, where relative position embeddings and the neighboring filter application are key factors. Based on these findings, we propose the enhanced local self-attention (ELSA) with Hadamard attention and the ghost head. Hadamard attention introduces the Hadamard product to efficiently generate attention in the neighboring case, while maintaining the high-order mapping. The ghost head combines attention maps with static matrices to increase channel capacity. Experiments demonstrate the effectiveness of ELSA. Without architecture / hyperparameter modification, drop-in replacing LSA with ELSA boosts Swin Transformer [48] by up to +1.4 on top-1 accuracy. ELSA also consistently benefits VOLO [83] from D1 to D5, where ELSA-VOLO-D5 achieves 87.2 on the ImageNet-1K without extra training images. In addition, we evaluate ELSA in downstream tasks. ELSA significantly improves the baseline by up to +1.9 box Ap / +1.3 mask Ap on the COCO, and by up to +1.9 mIoU on the ADE20K. Code is available at https://github.com/damo-cv/ELSA.

show abstract

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows

Cited by 75 publications

References 68 publications

QuadTree Attention for Vision Transformers

QuadTree Attention for Vision Transformers

MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition

ELSA: Enhanced Local Self-Attention for Vision Transformer

Contact Info

Product

Resources

About