“…The Vision Transformer (ViT) architecture is firstly proposed in (Dosovitskiy et al, 2020), which uses the attention mechanism (Vaswani et al, 2017) to solve various vision tasks. Compared to traditional CNN structures that operate on a fixed-sized window with restricted spatial interactions (Raghu et al, 2021), ViT allows all the positions in an image to interact through transformer blocks. Since then, many variants have been proposed (Graham et al, 2021;Liu et al, 2021c;Yuan et al, 2021a;Wang et al, 2021b;Han et al, 2021;Wu et al, 2021;Chen et al, 2021b;Steiner et al, 2021;El-Nouby et al, 2021;Liu et al, 2021a;Wang et al, 2021a;Bao et al, 2021).…”