EfficientViT: Enhanced Linear Attention for High-Resolution Low-Computation Visual Recognition

Abstract

Vision Transformer (ViT) has achieved remarkable performance in many vision tasks. However, ViT is inferior to convolutional neural networks (CNNs) when targeting high-resolution mobile vision applications. The key computational bottleneck of ViT is the softmax attention module which has quadratic computational complexity with the input resolution. It is essential to reduce the cost of ViT to deploy it on edge devices. Existing methods (e.g., Swin, PVT) restrict the softmax attention within local windows or reduce the resolution of key/value tensors to reduce the cost, which sacrifices ViT's core advantages on global feature extractions. In this work, we present EfficientViT, an efficient ViT architecture for high-resolution low-computation visual recognition. Instead of restricting the softmax attention, we propose to replace softmax attention with linear attention while enhancing its local feature extraction ability with depthwise convolution. EfficientViT maintains global and local feature extraction capability while enjoying linear computational complexity. Extensive experiments on COCO object detection and Cityscapes semantic segmentation demonstrate the effectiveness of our method. On the COCO dataset, EfficientViT achieves 42.6 AP with 4.4G MACs, surpassing EfficientDet-D1 by 2.4 AP while having 27.9% fewer MACs. On Cityscapes, EfficientViT reaches 78.7 mIoU with 19.1G MACs, outperforming SegFormer by 2.5 mIoU while requiring less than 1/3 the computational cost. On Qualcomm Snapdragon 855 CPU, EfficientViT is 3× faster than EfficientNet while achieving higher ImageNet accuracy.Preprint. Under review.

Linear Video Transformer with Feature Fixation

“…Despite the lower computational complexity, linear attention always presents a clear performance drop compared to Softmax [8,74,64]. We also observe this drop when applying the linear attention to video Transformers, as shown in Table 1 and 2.…”

“…Since we always have N D in practice, O(N D 2 ) is approximately equal to O(N ), i.e., growing linearly with the sequence length. As discussed in [64,8], a simple ReLU [1] is a good candidate for the kernel function, which satisfies the requirement of being non-negative and easily decomposable. The attended output is formulated with row-wise normalization, i.e.,…”

“…We also observe this drop when applying the linear attention to video Transformers, as shown in Table 1 and 2. The lack of non-linear attention concentration in the process of dot-product normalization is considered to be the primary cause [64,8]. From Fig.…”

GradViT: Gradient Inversion of Vision Transformers

“…Boosting up ViT efficiency has therefore been a very vibrant area. One stream of approach roots in the efficient deep learning literature that cuts down on network complexity leveraging popular methods such as efficient attention [3,41,4], network compression [7,8,34,67], dynamic inference [69,48], operator adaptation [43], token merging and manipulations [42,66], etc. These methods can yield off-the-shelf speedups on target ViT backbones, but are also limited to the original backbone's accuracy and capacity.…”

“…We also avoid squeeze-andexcitation operators and minimize Layer Normalization for higher resolution stages (i.e., 1, 2), as these layers tend to be math-limited. Later stages (i.e., 3,4) in the architecture tend to be math-limited as GPU hardware spends more time on compute compared to the memory transfer cost. As a result, applying multi-head attention will not be a bottleneck.…”