Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification

Wang, Yulin; Lv, Kangchen; Huang, Rui; Song, Shiji; Yang, Le; Huang, Gao

doi:10.48550/arxiv.2010.05300

Cited by 16 publications

(6 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been a great number of works help models focus on important features and adopt geometric variations in computer vision. All related works fall into two categories, attention-based methods [1,13,14,27] and offset-based methods [5,8,28,34,35]. We mainly review offset-based ones.…”

Section: Deformable-related Workmentioning

confidence: 99%

“…Unlike our task, region proposal network uses supervision of bounding box annotations. In image classification task, there are also some works explicitly learning positions of the important regions for better performance [8] or faster inference [28]. The learning process are merely guided by cross-entropy loss and final accuracy.…”

Section: Deformable-related Workmentioning

confidence: 99%

See 1 more Smart Citation

DPT: Deformable Patch-based Transformer for Visual Recognition

Zhi-yang

Zhu

Zhao

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Transformer has achieved great success in computer vision, while how to split patches in an image remains a problem. Existing methods usually use a fixed-size patch embedding which might destroy the semantics of objects. To address this problem, we propose a new Deformable Patch (DePatch) module which learns to adaptively split the images into patches with different positions and scales in a data-driven way rather than using predefined fixed patches. In this way, our method can well preserve the semantics in patches. The DePatch module can work as a plug-and-play module, which can easily be incorporated into different transformers to achieve an end-to-end training. We term this DePatch-embedded transformer as Deformable Patch-based Transformer (DPT) and conduct extensive evaluations of DPT on image classification and object detection. Results show DPT can achieve 81.9% top-1 accuracy on ImageNet classification, and 43.7% box mAP with RetinaNet, 44.3% with Mask R-CNN on MSCOCO object detection. Code has been made available at: https://github.com/CASIA-IVA-Lab/DPT.

show abstract

Section: Deformable-related Workmentioning

confidence: 99%

Section: Deformable-related Workmentioning

confidence: 99%

DPT: Deformable Patch-based Transformer for Visual Recognition

Zhi-yang

Zhu

Zhao

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…However, computational efficiency is critical in real-world scenarios, where the executed computation is translated into power consumption or carbon emission. Many works have tried on reducing the computational cost of CNNs via neural architecture search [10,16,25,54,57], knowledge distillation [20,55], dynamic routing [4,13,43,51,56] and pruning [15,18], but how to accelerate the ViT model have been rarely explored.…”

Section: Model Compressionmentioning

confidence: 99%

Make A Long Image Short: Adaptive Token Length for Vision Transformers

Zhu¹,

Zhu²,

Du³

et al. 2021

Preprint

View full text Add to dashboard Cite

The vision transformer splits each image into a sequence of tokens with fixed length and processes the tokens in the same way as words in natural language processing. More tokens normally lead to better performance but considerably increased computational cost. Motivated by the proverb "A picture is worth a thousand words" we aim to accelerate the ViT model by making a long image short. To this end, we propose a novel approach to assign token length adaptively during inference. Specifically, we first train a ViT model, called Resizable-ViT (ReViT), that can process any given input with diverse token lengths. Then, we retrieve the "token-length label" from ReViT and use it to train a lightweight Token-Length Assigner (TLA). The token-length labels are the smallest number of tokens to split an image that the ReViT can make the correct prediction, and TLA is learned to allocate the optimal token length based on these labels. The TLA enables the ReViT to process the image with the minimum sufficient number of tokens during inference. Thus, the inference speed is boosted by reducing the token numbers in the ViT model. Our approach is general and compatible with modern vision transformer architectures and can significantly reduce computational expanse. We verified the effectiveness of our methods on multiple representative ViT models (DeiT [39], LV-ViT [22], and TimesFormer [3]) across two tasks (image classification and action recognition).

show abstract

“…These adaptive computation strategies help speed up the inference time of convolutional neural networks (CNNs) [32], recurrent neural network (RNNs) [14], and also self-attention based methods (BERT) [24]. Besides the model-level adaptation, others further extend this idea to data-level adaptation, by either reducing the spatial redundancy [56] or focusing on key area [52]. However, those methods are limited by the convolutional structure, where only 2D data can be taken as input.…”

Section: Related Workmentioning

confidence: 99%

IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Pan¹,

Jiang²,

Panda³

et al. 2021

Preprint

View full text Add to dashboard Cite

The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory cost. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED 2 ). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4× speed-up for state-of-the-art models like DeiT [47] and TimeSformer [3], by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.Preprint. Under review.

show abstract

Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification

Cited by 16 publications

References 38 publications

DPT: Deformable Patch-based Transformer for Visual Recognition

DPT: Deformable Patch-based Transformer for Visual Recognition

Make A Long Image Short: Adaptive Token Length for Vision Transformers

IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Contact Info

Product

Resources

About