Adaptive Token Sampling For Efficient Vision Transformers

Abstract

While state-of-the-art vision transformer models achieve promising results for image classification, they are computationally very expensive and require many GFLOPs. Although the GFLOPs of a vision transformer can be decreased by reducing the number of tokens in the network, there is no setting that is optimal for all input images. In this work, we therefore introduce a differentiable parameterfree Adaptive Token Sampling (ATS) module, which can be plugged into any existing vision transformer architecture. ATS empowers vision transformers by scoring and adaptively sampling significant tokens. As a result, the number of tokens is not anymore static but it varies for each input image. By integrating ATS as an additional layer within current transformer blocks, we can convert them into much more efficient vision transformers with an adaptive number of tokens. Since ATS is a parameter-free module, it can be added to off-the-shelf pretrained vision transformers as a plug and play module, thus reducing their GFLOPs without any additional training. However, due to its differentiable design, one can also train a vision transformer equipped with ATS. We evaluate our module on the ImageNet dataset by adding it to multiple state-of-the-art vision transformers. Our evaluations show that the proposed module improves the state-of-the-art by reducing the computational cost (GFLOPs) by 37% while preserving the accuracy.

SPViT: Enabling Faster Vision Transformers via Soft Token Pruning

“…Main Results. We compare our method with several representative methods including DynamicViT [53], IA-RED 2 [48], RegNetY [51], CrossViT [6], VTP [86], ATS [24], CvT [66], PVT [64], T2T-ViT [79], UP-DeiT [76], PS-ViT [59], Evo-ViT [70], TNT [29], HVT [49], Swin [43], CoaT [69], CPVT [16], MD-DeiT [34], and S 2 ViTE [11]. As shown in Table 14 produced model is 0.91% higher than the original PiT-XS; When the target size is PiT-T, the accuracy of the produced model is 0.9% higher than the original PiT-T.…”

EfficientFormer: Vision Transformers at MobileNet Speed

“…Despite the success in most vision tasks, ViT-based models cannot compete with the well-studied lightweight CNNs [21,49] when the inference speed is the major concern [50,51,52], especially on resource-constrained edge devices [17]. To accelerate ViT, many approaches have been introduced with different methodologies, such as proposing new architectures or modules [53,54,55,56,57,58], re-thinking self-attention and sparse-attention mechanisms [59,60,61,62,63,64,65], and utilizing search algorithms that are widely explored in CNNs to find smaller and faster ViTs [66,28,29,67]. Recently, LeViT [23] proposes a CONV-clothing design to accelerate vision transformer.…”

SimA: Simple Softmax-free Attention for Vision Transformers

“…[38] uses distillation to improve the efficiency of the network. [14,46,39] decrease the number of tokens by pruning unimportant tokens. Although these works limit the computation generally, softmax is still required to calculate the attention.…”