ConTNet: Why not use convolution and transformer at the same time?

Yan, Haotian; Li, Zhe; Li, Weijian; Wang, Changhu; Wu, Ming; Zhang, Chuang

doi:10.48550/arxiv.2104.13497

Cited by 25 publications

(27 citation statements)

References 54 publications

(98 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DeiT [38] improves the data efficiency of training ViT with a token distillation pipeline. Apart from the sequence-to-sequence structure, the efficiency of PVT [39] and Swin Transformer [30] sparks much interests in exploring the Hierarchical Vision Transformer (HVT) [14,22,41,44]. ViT is also extended to solve the low-level tasks and dense prediction problems [2,6,20].…”

Section: Vision Transformermentioning

confidence: 99%

Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention

Yan¹,

Zhang²,

Wu³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Multi-scale representations are crucial for semantic segmentation. The community has witnessed the flourish of semantic segmentation convolutional neural networks (CNN) exploiting multi-scale contextual information. Motivated by that the vision transformer (ViT) is powerful in image classification, some semantic segmentation ViTs are recently proposed, most of them attaining impressive results but at a cost of computational economy. In this paper, we succeed in introducing multi-scale representations into semantic segmentation ViT via window attention mechanism and further improves the performance and efficiency. To this end, we introduce large window attention which allows the local window to query a larger area of context window at only a little computation overhead. By regulating the ratio of the context area to the query area, we enable the large window attention to capture the contextual information at multiple scales. Moreover, the framework of spatial pyramid pooling is adopted to collaborate with the large window attention, which presents a novel decoder named large window attention spatial pyramid pooling (LawinASPP) for semantic segmentation ViT. Our resulting ViT, Lawin Transformer, is composed of an efficient hierachical vision transformer (HVT) as encoder and a LawinASPP as decoder. The empirical results demonstrate that Lawin Transformer offers an improved efficiency compared to the existing method. Lawin Transformer further sets new state-of-the-art performance on Cityscapes (84.4% mIoU), ADE20K (56.2% mIoU) and COCO-Stuff datasets. The code will be released at https://github.com/yan-hao-tian/lawin.

show abstract

Section: Vision Transformermentioning

confidence: 99%

Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention

Yan¹,

Zhang²,

Wu³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The recent advance [12] shows that the transformer can also achieve incredible performance on computer vision tasks. While vision transformer suffers from the necessity large-scaled dataset [44], many recent works try to encode strong inductive prior by either combining it with convolutional layer [53,31,57,55] or introducing 2D-hierarchical structure to vision transformer [33,49,13,9]. Besides, transformer also shows strong power in other vision tasks, including semantic segmentation [60], object detection [5,61], image processing [10], and image generation [27,26].…”

Section: Related Workmentioning

confidence: 99%

IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Pan¹,

Jiang²,

Panda³

et al. 2021

Preprint

View full text Add to dashboard Cite

The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory cost. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED 2 ). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4× speed-up for state-of-the-art models like DeiT [47] and TimeSformer [3], by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.Preprint. Under review.

show abstract

“…Moreover, the non-linear embedding method (SNE) also improves the classification rates on the ImageNet dataset, compared to the baseline model which uses the linear embedding method. ViT-Ti [8] 5.7M T 68.7 T2T-ViT-7 [37] 4.3M T 71.7 DeiT-Ti [29] 5.7M T 72.2 Mobile-Former-96M [3] 4.6M C+T 72.8 LocalViT-T2T [18] 4.3M C+T 72.5 PiT-Ti [11] 4.9M C+T 73.0 ConViT-Ti [5] 5.7M C+T 73.1 ConT-Ti [34] 5.8M C+T 74.9 LocalViT-T [18] 5.9M C+T 74.8 ViTAE-T [33] 4.8M C+T 75.3 EfficientNet-B0 [28] 5.3M C 76.3 CeiT-T [36] 6.4M C+T 76.4 T2T-ViT-12 [37] 6.9M T 76.5 ConT-S [34] 10.1M C+T 76.5 CoaT-Lite Tiny [32] 5.7M C+T 76.6 Swin-1G [3,19] 7.3M T 77.3 XCiT-T12 (baseline) [9] 6.7M T 77.…”

Section: Imagenet Classificationmentioning

confidence: 99%

Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints

Ahn¹,

Hong²,

Ju³

et al. 2021

Preprint

View full text Add to dashboard Cite

A vision transformer (ViT) is the dominant model in the computer vision field. Despite numerous studies that mainly focus on dealing with inductive bias and complexity, there remains the problem of finding better transformer networks. For example, conventional transformer-based models usually use a projection layer for each query (Q), key (K), and value (V) embedding before multi-head self-attention. Insufficient consideration of semantic Q, K, and V embedding may lead to a performance drop. In this paper, we propose three types of structures for Q, K, and V embedding. The first structure utilizes two layers with ReLU, which is a non-linear embedding for Q, K, and V . The second involves sharing one of the non-linear layers to share knowledge among Q, K, and V . The third proposed structure shares all non-linear layers with code parameters. The codes are trainable, and the values determine the embedding process to be performed among Q, K, and V . Hence, we demonstrate the superior image classification performance of the proposed approaches in experiments compared to several state-of-the-art approaches. The proposed method achieved 71.4% with a few parameters (of 3.1M ) on the ImageNet-1k dataset compared to that required by the original transformer model of . Additionally, the method achieved 93.3% with only 2.9M parameters in transfer learning on average for the CIFAR-10, CIFAR-100, Stanford Cars datasets, and STL-10 datasets, which is better than the accuracy of 92.2% obtained via the original XCiT-N12 model.

show abstract

ConTNet: Why not use convolution and transformer at the same time?

Cited by 25 publications

References 54 publications

Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention

Lawin Transformer: Improving Semantic Segmentation Transformer with Multi-Scale Representations via Large Window Attention

IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Rethinking Query, Key, and Value Embedding in Vision Transformer under Tiny Model Constraints

Contact Info

Product

Resources

About