MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens

Fang, Jiemin; Xie, Lingxi; Wang, Xinggang; Zhang, Xiaopeng; Liu, Wenyu

doi:10.48550/arxiv.2105.15168

Cited by 11 publications

(15 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Swin transformer [21] used a shifted window mechanism. Although the design served as an excellent backbone for the following works [3,5,10,13,38], the shifted window communication had an overhead of careful cyclic shift and padding. Shortly after, the work [5] proposed two models, namely Twins-PCPVT and Twins-SVT.…”

Section: Related Workmentioning

confidence: 99%

“…The global sub-sampled attention (GSA) module used strided convolution function to summarise the local tokens and apply MSA for global interaction. The MSGTr [13] and RegionViT [3] proposed a similar idea of interacting with a regional level representative tokens. We discuss these works in more detail in the next section to clarify the design differences.…”

Section: Related Workmentioning

confidence: 99%

“…3). The MSG-Transformer [13] has window messenger tokens that exchange information on regional level (group of nearby windows) by channelgroups shuffling. Thus, the purpose of a messenger token is only to exchange some feature channels with nearby windows.…”

Section: Complexity Analysismentioning

confidence: 99%

“…Moreover, each Super token also maintains its identity and connection to the specific image window. Region-ViT [3] also exploits regional and local level interaction as the MSG-Transformer [13]. However, different from the proposed STT, the regional tokens are coming from a parallel stream of transformer, having a bigger patch size than the local stream and initialised from data tokens.…”

Section: Complexity Analysismentioning

confidence: 99%

“…In CV, the number of data tokens is fairly high, due to the 2D spatial nature of the input. Several techniques have been proposed to address the computational complexity of transformers [3,5,12,13,16,18,21,29,32,35]. Window-based self-attention (WMSA) has become one of the most popular mechanisms to reduce computational complexity of the self-attention operation [5,21,32].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Global Interaction Modelling in Vision Transformer via Super Tokens

Farooq¹,

Awais²,

Ahmed³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the popularity of Transformer architectures in computer vision, the research focus has shifted towards developing computationally efficient designs. Window-based local attention is one of the major techniques being adopted in recent works. These methods begin with very small patch size and small embedding dimensions and then perform strided convolution (patch merging) in order to reduce the feature map size and increase embedding dimensions, hence, forming a pyramidal Convolutional Neural Network (CNN) like design. In this work, we investigate local and global information modelling in transformers by presenting a novel isotropic architecture that adopts local windows and special tokens, called Super tokens, for self-attention. Specifically, a single Super token is assigned to each image window which captures the rich local details for that window. These tokens are then employed for cross-window communication and global representation learning. Hence, most of the learning is independent of the image patches (N ) in the higher layers, and the class embedding is learned solely based on the Super tokens (N M 2 ) where M 2 is the window size. In standard image classification on Imagenet-1K, the proposed Super tokens based transformer (STT-S25) achieves 83.5% accuracy which is equivalent to Swin transformer (Swin-B) with circa half the number of parameters (49M) and double the inference time throughput. The proposed Super token transformer offers a lightweight and promising backbone for visual recognition tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Complexity Analysismentioning

confidence: 99%

Section: Complexity Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Global Interaction Modelling in Vision Transformer via Super Tokens

Farooq¹,

Awais²,

Ahmed³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

What Makes for Hierarchical Vision Transformer?

Fang¹,

Wang

Wu³

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

Tang

Zhao

Wang

et al. 2022

AAAI

View full text Add to dashboard Cite

Transformers have sprung up in the field of computer vision. In this work, we explore whether the core self-attention module in Transformer is the key to achieving excellent performance in image recognition. To this end, we build an attention-free network called sMLPNet based on the existing MLP-based vision models. Specifically, we replace the MLP module in the token-mixing step with a novel sparse MLP (sMLP) module. For 2D image tokens, sMLP applies 1D MLP along the axial directions and the parameters are shared among rows or columns. By sparse connection and weight sharing, sMLP module significantly reduces the number of model parameters and computational complexity, avoiding the common over-fitting problem that plagues the performance of MLP-like models. When only trained on the ImageNet-1K dataset, the proposed sMLPNet achieves 81.9% top-1 accuracy with only 24M parameters, which is much better than most CNNs and vision Transformers under the same model size constraint. When scaling up to 66M parameters, sMLPNet achieves 83.4% top-1 accuracy, which is on par with the state-of-the-art Swin Transformer. The success of sMLPNet suggests that the self-attention mechanism is not necessarily a silver bullet in computer vision. The code and models are publicly available at https://github.com/microsoft/SPACH.

show abstract

MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens

Cited by 11 publications

References 51 publications

Global Interaction Modelling in Vision Transformer via Super Tokens

Global Interaction Modelling in Vision Transformer via Super Tokens

What Makes for Hierarchical Vision Transformer?

Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?

Contact Info

Product

Resources

About