A-ViT: Adaptive Tokens for Efficient Vision Transformer

Unmanned aerial vehicles (UAVs) are becoming popular in various applications. However, there are still challenging issues to be tackled, such as effective obstacle avoidance, target identification within a crowd, and specific target tracking. This paper focuses on dynamic target following and obstacle avoidance to realize a prototype of a quadcopter drone to serve as an autonomous object follower. An adaptive target identification system is proposed to recognize the specific target in the complicated background. For obstacle avoidance during flight, we introduce an idea of space detection and use it to develop a so-called contour and spiral convolution space detection (CASCSD) algorithm to evade obstacles. Thanks to the low architecture complexity, it is appropriate for implementation on onboard flight control systems. The target prediction is integrated with fuzzified flight control to fulfill an autonomous target tracker. When this series of technical research and development is completed, this system can be used for applications such as personal security guard and criminal detection systems.

show abstract

“…• A-ViT [22]: A-ViT uses an adaptive token discarding framework to expedite inference in vision transformers.…”

Section: B Comparison With Sota Methodsmentioning

confidence: 99%

“…Evo-ViT [21] uses a slow-fast token evolution strategy to update the attractive and unattractive patch groups. A-ViT [22] augments the vision transformer block with adaptive halting modules that compute a halting probability to prune unnecessary tokens.…”

Section: B Patch Slimmingmentioning

confidence: 99%

Quadcopter Drone for Vision-Based Autonomous Target Following

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Various techniques have been developed to train ViT efficiently. Among them, token sparsification (Pan et al, 2021;Rao et al, 2021;Tang et al, 2022;Yin et al, 2022) removes redundant tokens (image patches) of data to improve the computational complexity while maintaining a comparable learning performance. For example, Under what conditions does a Transformer achieve satisfactory generalization?…”

Section: Introductionmentioning

confidence: 99%

A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity

Hong-kang¹,

Wang²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Vision Transformers (ViTs) with self-attention modules have recently achieved great empirical success in many vision tasks. Due to non-convex interactions across layers, however, the theoretical learning and generalization analysis is mostly elusive. Based on a data model characterizing both label-relevant and label-irrelevant tokens, this paper provides the first theoretical analysis of training a shallow ViT, i.e., one self-attention layer followed by a two-layer perceptron, for a classification task. We characterize the sample complexity to achieve a zero generalization error. Our sample complexity bound is positively correlated with the inverse of the fraction of label-relevant tokens, the token noise level, and the initial model error. We also prove that a training process using stochastic gradient descent (SGD) leads to a sparse attention map, which is a formal verification of the general intuition about the success of attention. Moreover, this paper indicates that a proper token sparsification can improve the test performance by removing label-irrelevant and/or noisy tokens, including spurious correlations. Empirical experiments on synthetic data and CIFAR-10 dataset justify our theoretical results and generalize to deeper ViTs.

show abstract

“…(3) We demonstrate the efficacy of GradMDM on multiple dynamic neural networks across various datasets, where GradMDM effectively increases computations with less perceptible perturbations across all settings. [19], [20], [21], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33] achieve efficiency while maintaining good accuracy by dynamically adjusting model architectures to allocate appropriate computation conditioned on each input sample. This reduces redundant computations on those "easy" samples, improving the inference efficiency [18].…”

Section: Introductionmentioning

confidence: 99%

“…SkipNet [19] assigns a gating module to each convolutional block in CNNs, which decides whether to execute or skip it, and is trained via reinforcement learning. Dynamic width networks [21], [32], [34] selectively activate multiple components within the arXiv:2304.06724v1 [cs.CR] 1 Apr 2023 same layer, such as channels and neurons, based on each instance. Early studies [35], [36], [37] achieve dynamic width by adaptively controlling the activation of neurons or parameters, e.g., via stochastic gating units [35], [36].…”

Section: Introductionmentioning

confidence: 99%

GradMDM: Adversarial Attack on Dynamic Networks

Pan¹,

Foo²,

Zheng³

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Dynamic neural networks can greatly reduce computation redundancy without compromising accuracy by adapting their structures based on the input. In this paper, we explore the robustness of dynamic neural networks against energy-oriented attacks targeted at reducing their efficiency. Specifically, we attack dynamic models with our novel algorithm GradMDM. GradMDM is a technique that adjusts the direction and the magnitude of the gradients to effectively find a small perturbation for each input, that will activate more computational units of dynamic models during inference. We evaluate GradMDM on multiple datasets and dynamic models, where it outperforms previous energy-oriented attack techniques, significantly increasing computation complexity while reducing the perceptibility of the perturbations.

show abstract

A-ViT: Adaptive Tokens for Efficient Vision Transformer

Cited by 125 publications

References 19 publications

Quadcopter Drone for Vision-Based Autonomous Target Following

Quadcopter Drone for Vision-Based Autonomous Target Following

A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity

GradMDM: Adversarial Attack on Dynamic Networks

Contact Info

Product

Resources

About