PSViT: Better Vision Transformer via Token Pooling and Attention Sharing

Chen, Boyu; Li, Peixia; Li, Baopu; Li, Chuming; Bai, Lei; Lin, Chen; Sun, Ming; Yan, Junjie; Ouyang, Wanli

doi:10.48550/arxiv.2108.03428

Cited by 5 publications

(6 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, such unstructured sparsity results in incompatibility with dense prediction tasks. Some structurepreserving token selection strategies were implemented via token pooling (Chen et al 2021a) and a slow-fast updating (Xu et al 2021).…”

Section: Efficient Vision Transformersmentioning

confidence: 99%

Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention

Tan

et al. 2022

AAAI

View full text Add to dashboard Cite

Recently, Transformers have shown promising performance in various vision tasks. To reduce the quadratic computation complexity caused by the global self-attention, various methods constrain the range of attention within a local region to improve its efficiency. Consequently, their receptive fields in a single attention layer are not large enough, resulting in insufficient context modeling. To address this issue, we propose a Pale-Shaped self-Attention (PS-Attention), which performs self-attention within a pale-shaped region. Compared to the global self-attention, PS-Attention can reduce the computation and memory costs significantly. Meanwhile, it can capture richer contextual information under the similar computation complexity with previous local self-attention mechanisms. Based on the PS-Attention, we develop a general Vision Transformer backbone with a hierarchical architecture, named Pale Transformer, which achieves 83.4%, 84.3%, and 84.9% Top-1 accuracy with the model size of 22M, 48M, and 85M respectively for 224x224 ImageNet-1K classification, outperforming the previous Vision Transformer backbones. For downstream tasks, our Pale Transformer backbone performs better than the recent state-of-the-art CSWin Transformer by a large margin on ADE20K semantic segmentation and COCO object detection & instance segmentation. The code will be released on https://github.com/BR-IDL/PaddleViT.

show abstract

Section: Efficient Vision Transformersmentioning

confidence: 99%

Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention

Tan

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Many subsequent works address this issue by establishing a progressive shrinking pyramid that allows models to explicitly process low-level patterns. There is a group of approaches that merge tokens within each fixed window into one to reduce the number of tokens [8,24,37,46,63,66,76]. In contrast, the second group of methods drops this constraint and introduces more flexible selection scheme [9,43,50,77].…”

Section: Related Workmentioning

confidence: 99%

“…However, other tokens maintain the ability to express distinctive patterns and may delicately assist final prediction. Some works proposed to remove the [cls] token and construct a global token by integrating patch tokens via certain average pooling operation [8,37,46,51]. LV-ViT [29] explored the possibility to jointly utilize [cls] token and patch tokens.…”

Section: Related Workmentioning

confidence: 99%

Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

He¹,

Liang²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents new hierarchically cascaded transformers that can improve data efficiency through attribute surrogates learning and spectral tokens pooling. Vision transformers have recently been thought of as a promising alternative to convolutional neural networks for visual recognition. But when there is no sufficient data, it gets stuck in overfitting and shows inferior performance. To improve data efficiency, we propose hierarchically cascaded transformers that exploit intrinsic image structures through spectral tokens pooling and optimize the learnable parameters through latent attribute surrogates. The intrinsic image structure is utilized to reduce the ambiguity between foreground content and background noise by spectral tokens pooling. And the attribute surrogate learning scheme is designed to benefit from the rich visual information in image-label pairs instead of simple visual concepts assigned by their labels. Our Hierarchically Cascaded Transformers, called HCTransformers, is built upon a self-supervised learning framework DINO and is tested on several popular few-shot learning benchmarks.In the inductive setting, HCTransformers surpass the DINO baseline by a large margin of 9.7% 5-way 1shot accuracy and 9.17% 5-way 5-shot accuracy on miniImageNet, which demonstrates HCTransformers are efficient to extract discriminative features. Also, HCTransformers show clear advantages over SOTA few-shot classification methods in both 5-way 1-shot and 5-way 5-shot settings on four popular benchmark datasets, including miniImageNet, tieredImageNet, FC100, and CIFAR-FS. The trained weights and codes are available at https: //github.com/StomachCold/HCTransformers.

show abstract

“…Recently, the pioneering work ViT [22] successfully applies the pure transformer-based architecture to computer vision, revealing the potential of transformer in handling visual tasks. Lots of follow-up studies are proposed [4,5,9,12,18,21,23,24,[27][28][29]31,38,41,43,45,50,52,56,76,77,80,81,84]. Many of them analyze the ViT [15,17,26,32,44,55,69,73,75,82] and improve it via introducing locality to earlier layers [11,17,48,64,79,83,87].…”

Section: Related Workmentioning

confidence: 99%

ELSA: Enhanced Local Self-Attention for Vision Transformer

Zhou¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-attention is powerful in modeling long-range dependencies, but it is weak in local finer-level feature learning. The performance of local self-attention (LSA) is just on par with convolution and inferior to dynamic filters, which puzzles researchers on whether to use LSA or its counterparts, which one is better, and what makes LSA mediocre. To clarify these, we comprehensively investigate LSA and its counterparts from two sides: channel setting and spatial processing. We find that the devil lies in the generation and application of spatial attention, where relative position embeddings and the neighboring filter application are key factors. Based on these findings, we propose the enhanced local self-attention (ELSA) with Hadamard attention and the ghost head. Hadamard attention introduces the Hadamard product to efficiently generate attention in the neighboring case, while maintaining the high-order mapping. The ghost head combines attention maps with static matrices to increase channel capacity. Experiments demonstrate the effectiveness of ELSA. Without architecture / hyperparameter modification, drop-in replacing LSA with ELSA boosts Swin Transformer [48] by up to +1.4 on top-1 accuracy. ELSA also consistently benefits VOLO [83] from D1 to D5, where ELSA-VOLO-D5 achieves 87.2 on the ImageNet-1K without extra training images. In addition, we evaluate ELSA in downstream tasks. ELSA significantly improves the baseline by up to +1.9 box Ap / +1.3 mask Ap on the COCO, and by up to +1.9 mIoU on the ADE20K. Code is available at https://github.com/damo-cv/ELSA.

show abstract

PSViT: Better Vision Transformer via Token Pooling and Attention Sharing

Cited by 5 publications

References 25 publications

Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention

Pale Transformer: A General Vision Transformer Backbone with Pale-Shaped Attention

Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning

ELSA: Enhanced Local Self-Attention for Vision Transformer

Contact Info

Product

Resources

About