What Makes for Hierarchical Vision Transformer?

Abstract

Recent studies show that hierarchical Vision Transformer with interleaved non-overlapped intra window selfattention & shifted window self-attention is able to achieve state-of-the-art performance in various visual recognition tasks and challenges CNN's dense sliding window paradigm. Most follow-up works try to replace shifted window operation with other kinds of cross window communication while treating self-attention as the de-facto standard for intra window information aggregation. In this short preprint, we question whether self-attention is the only choice for hierarchical Vision Transformer to attain strong performance, and what makes for hierarchical Vision Transformer? We replace self-attention layers in Swin Transformer and Shuffle Transformer with simple linear mapping and keep other components unchanged.The resulting architecture with 25.4M parameters and 4.2G FLOPs achieves 80.5% Top-1 accuracy, compared to 81.3% for Swin Transformer with 28.3M parameters and 4.5G FLOPs. We also experiment with other alternatives to self-attention for context aggregation inside each non-overlapped window, which all give similar competitive results under the same architecture. Our study reveals that the macro architecture of Swin model families (i.e., interleaved intra window & cross window communications), other than specific aggregation layers or specific means of cross window communication, may be more responsible for its strong performance and is the real challenger to CNN's dense sliding window paradigm.

ELSA: Enhanced Local Self-Attention for Vision Transformer

“…LSA only achieves similar top-1 accuracy as DwConv, which is lower than DDF, but it requires more floating-point operations (FLOPs). This phenomenon has also been observed in recent papers [17,25,32,83], but they lack detailed analysis of the reason behind such performances, and it motivates us to raise a question: what makes local self-attention mediocre?…”

“…Swin Transformer [48], as a milestone, also leverages local self-attention (LSA) to embed detailed information in high-resolution finer-level features. Despite these successes, several studies [17,25,32,83] observe that the performance of LSA is just on par with convolution in both upstream and downstream tasks [32]. The reasons behind this phenomenon are not clear, and in-depth comparisons under the same conditions are valuable.…”