MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

Mehta, Sachin; Rastegari, Mohammad

doi:10.48550/arxiv.2110.02178

Cited by 116 publications

(177 citation statements)

References 20 publications

Supporting

Mentioning

166

Contrasting

Order By: Relevance

“…The Cross-ViT (Chen et al, 2021a) uses a dual-branch transformer to combine image patches of different sizes to produce stronger image features, and propose a cross attention module to reduce computation. The MobileViT (Mehta & Rastegari, 2021) combines the strengths of CNNs and ViTs by replacing local processing in convolutions with global processing using transformers.…”

Section: Lightweight Vitsmentioning

confidence: 99%

See 1 more Smart Citation

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

Sun¹,

Ma²,

Kang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The transformer architectures with attention mechanisms have obtained success in Nature Language Processing (NLP), and Vision Transformers (ViTs) have recently extended the application domains to various vision tasks. While achieving high performance, ViTs suffer from large model size and high computation complexity that hinders the deployment of them on edge devices. To achieve high throughput on hardware and preserve the model accuracy simultaneously, we propose VAQF, a framework that builds inference accelerators on FPGA platforms for quantized ViTs with binary weights and low-precision activations. Given the model structure and the desired frame rate, VAQF will automatically output the required quantization precision for activations as well as the optimized parameter settings of the accelerator that fulfill the hardware requirements. The implementations are developed with Vivado High-Level Synthesis (HLS) on the Xilinx ZCU102 FPGA board, and the evaluation results with the DeiT-base model indicate that a frame rate requirement of 24 frames per second (FPS) is satisfied with 8-bit activation quantization, and a target of 30 FPS is met with 6-bit activation quantization. To the best of our knowledge, this is the first time quantization has been incorporated into ViT acceleration on FPGAs with the help of a fully automatic framework to guide the quantization strategy on the software side and the accelerator implementations on the hardware side given the target frame rate. Very small compilation time cost is incurred compared with quantization training, and the generated accelerators show the capability of achieving real-time execution for state-of-the-art ViT models on FPGAs.

show abstract

Section: Lightweight Vitsmentioning

confidence: 99%

“…More details of the efficiency evaluation will be discussed in Section 6.3. 72.2 5.7M × 32 PiT (Heo et al, 2021) 73.0 4.9M × 32 Cross-ViT (Chen et al, 2021a) 73.4 6.9M × 32 MobileViT (Mehta & Rastegari, 2021) 74.8 2.3M × 32…”

Section: Comparison With Other Lightweight Vitsmentioning

confidence: 99%

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

Sun¹,

Ma²,

Kang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, the pioneering work ViT [22] successfully applies the pure transformer-based architecture to computer vision, revealing the potential of transformer in handling visual tasks. Lots of follow-up studies are proposed [4,5,9,12,18,21,23,24,[27][28][29]31,38,41,43,45,50,52,56,76,77,80,81,84]. Many of them analyze the ViT [15,17,26,32,44,55,69,73,75,82] and improve it via introducing locality to earlier layers [11,17,48,64,79,83,87].…”

Section: Related Workmentioning

confidence: 99%

ELSA: Enhanced Local Self-Attention for Vision Transformer

Zhou¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-attention is powerful in modeling long-range dependencies, but it is weak in local finer-level feature learning. The performance of local self-attention (LSA) is just on par with convolution and inferior to dynamic filters, which puzzles researchers on whether to use LSA or its counterparts, which one is better, and what makes LSA mediocre. To clarify these, we comprehensively investigate LSA and its counterparts from two sides: channel setting and spatial processing. We find that the devil lies in the generation and application of spatial attention, where relative position embeddings and the neighboring filter application are key factors. Based on these findings, we propose the enhanced local self-attention (ELSA) with Hadamard attention and the ghost head. Hadamard attention introduces the Hadamard product to efficiently generate attention in the neighboring case, while maintaining the high-order mapping. The ghost head combines attention maps with static matrices to increase channel capacity. Experiments demonstrate the effectiveness of ELSA. Without architecture / hyperparameter modification, drop-in replacing LSA with ELSA boosts Swin Transformer [48] by up to +1.4 on top-1 accuracy. ELSA also consistently benefits VOLO [83] from D1 to D5, where ELSA-VOLO-D5 achieves 87.2 on the ImageNet-1K without extra training images. In addition, we evaluate ELSA in downstream tasks. ELSA significantly improves the baseline by up to +1.9 box Ap / +1.3 mask Ap on the COCO, and by up to +1.9 mIoU on the ADE20K. Code is available at https://github.com/damo-cv/ELSA.

show abstract

“…Additionally, the vanilla ViT models and, especially, their larger variants, are very hard to train and require to be trained on huge annotated datasets. Following ViT, many methods [16,14,9,18,19] appeared trying to solve or circumvent these issues while also maintaining SOTA performance [4,12]. In this work, we incorporate two very promising architectures, DEIT [12] and Xcit [4] in our models and evaluate their performance.…”

Section: Vision Transformersmentioning

confidence: 99%

PE-former: Pose Estimation Transformer

Panteleris¹,

Argyros²

2021

Preprint

View full text Add to dashboard Cite

Vision transformer architectures have been demonstrated to work very effectively for image classification tasks. Efforts to solve more challenging vision tasks with transformers rely on convolutional backbones for feature extraction. In this paper we investigate the use of a pure transformer architecture (i.e., one with no CNN backbone) for the problem of 2D body pose estimation. We evaluate two ViT architectures on the COCO dataset. We demonstrate that using an encoder-decoder transformer architecture yields state of the art results on this estimation problem.

show abstract

MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

Cited by 116 publications

References 20 publications

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

ELSA: Enhanced Local Self-Attention for Vision Transformer

PE-former: Pose Estimation Transformer

Contact Info

Product

Resources

About