IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Pan, Bowen; Jiang, Yifan; Panda, Rameswar; Wang, Zhangyang; Feris, Rogério; Oliva, Aude

doi:10.48550/arxiv.2106.12620

Cited by 4 publications

(1 citation statement)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are two major approaches: unstructured token sparsification and structured token division. The majority of works, including PatchSlim [41], TokenSparse [36], GlobalEncoder [39], IA-RED [32], and Tokenlearner [37], focus on the former. TokenLearner [37] uses an MLP to reduce the number of tokens.…”

Section: Dynamic Vision Transformermentioning

confidence: 99%

Optimization of Water-Energy-Carbon Nexus in Urban Residential Water Uses for Shanghai, China

Zhou¹,

Zhu²

2023

Preprint

View full text Add to dashboard Cite

<p>Urban residential water uses entail energy consumption and associated carbon emissions. Reducing residential water uses thus can simultaneously save water and energy and help reduce carbon emissions. However, residential water uses are strongly affected by the choices of household appliances and fixtures, and water use behaviors. In this study, we first conducted a household water use survey in Shanghai, the largest city in China, to understand residential water use behaviors in different seasons. A two-stage stochastic optimization model is developed to optimize water and energy conservation decisions so as to minimize the expected total cost of water and water-related energy uses and maximize carbon emission reduction. Data collected through questionnaire surveys are used to parameterize the optimization model. Water and energy conservation choices are categorized into long- and short-term decisions. The results show that in Shanghai residential water uses has a strong impact on urban carbon emission reduction. Typical temperature range in different seasons strongly affects the effectiveness of short-term conservation actions. The results support a subsidy policy for water-saving appliances that can incentivize citizens in water-saving. Model results are useful for exploring the water-energy-carbon nexus of urban households considering seasonal factors</p>

show abstract

Section: Dynamic Vision Transformermentioning

confidence: 99%

Optimization of Water-Energy-Carbon Nexus in Urban Residential Water Uses for Shanghai, China

Zhou¹,

Zhu²

2023

Preprint

View full text Add to dashboard Cite

show abstract

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

Zhang

et al. 2022

AAAI

View full text Add to dashboard Cite

Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue. Since the computation complexity of ViT is quadratic with respect to the input sequence length, a mainstream paradigm for computation reduction is to reduce the number of tokens. Existing designs include structured spatial compression that uses a progressive shrinking pyramid to reduce the computations of large feature maps, and unstructured token pruning that dynamically drops redundant tokens. However, the limitation of existing token pruning lies in two folds: 1) the incomplete spatial structure caused by pruning is not compatible with structured spatial compression that is commonly used in modern deep-narrow transformers; 2) it usually requires a time-consuming pre-training procedure. To tackle the limitations and expand the applicable scenario of token pruning, we present Evo-ViT, a self-motivated slow-fast token evolution approach for vision transformers. Specifically, we conduct unstructured instance-wise token selection by taking advantage of the simple and effective global class attention that is native to vision transformers. Then, we propose to update the selected informative tokens and uninformative tokens with different computation paths, namely, slow-fast updating. Since slow-fast updating mechanism maintains the spatial structure and information flow, Evo-ViT can accelerate vanilla transformers of both flat and deep-narrow structures from the very beginning of the training process. Experimental results demonstrate that our method significantly reduces the computational cost of vision transformers while maintaining comparable performance on image classification. For example, our method accelerates DeiT-S by over 60% throughput while only sacrificing 0.4% top-1 accuracy on ImageNet-1K, outperforming current token pruning methods on both accuracy and efficiency.

show abstract

Efficient Video Transformers with Spatial-Temporal Token Selection

Wang¹,

Yang²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Video transformers have achieved impressive results on major video recognition benchmarks, however they suffer from high computational cost. In this paper, we present STTS, a token selection framework that dynamically selects a few informative tokens in both temporal and spatial dimensions conditioned on input video samples. Specifically, we formulate token selection as a ranking problem, which estimates the importance of each token through a lightweight selection network and only those with top scores will be used for downstream evaluation. In the temporal dimension, we keep the frames that are most relevant for recognizing action categories, while in the spatial dimension, we identify the most discriminative region in feature maps without affecting spatial context used in a hierarchical way in most video transformers. Since the decision of token selection is non-differentiable, we employ a perturbed-maximum based differentiable Top-K operator for end-to-end training. We conduct extensive experiments on Kinetics-400 with a recently introduced video transformer backbone, MViT. Our framework achieves similar results while requiring 20% less computation. We also demonstrate that our approach is compatible with other transformer architectures.

show abstract

IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

Cited by 4 publications

References 68 publications

Optimization of Water-Energy-Carbon Nexus in Urban Residential Water Uses for Shanghai, China

Optimization of Water-Energy-Carbon Nexus in Urban Residential Water Uses for Shanghai, China

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

Efficient Video Transformers with Spatial-Temporal Token Selection

Contact Info

Product

Resources

About