QuadTree Attention for Vision Transformers

Abstract

Transformers have been successful in many vision tasks, thanks to their capability of capturing long-range dependency. However, their quadratic computational complexity poses a major obstacle for applying them to vision tasks requiring dense predictions, such as object detection, feature matching, stereo, etc. We introduce QuadTree Attention, which reduces the computational complexity from quadratic to linear. Our quadtree transformer builds token pyramids and computes attention in a coarse-to-fine manner. At each level, the top K patches with the highest attention scores are selected, such that at the next level, attention is only evaluated within the relevant regions corresponding to these top K patches. We demonstrate that quadtree attention achieves state-of-theart performance in various vision tasks, e.g. with 4.0% improvement in feature matching on ScanNet, about 50% flops reduction in stereo matching, 0.4-1.5% improvement in top-1 accuracy on ImageNet classification, 1.2-1.8% improvement on COCO object detection, and 0.7-2.4% improvement on semantic segmentation over previous state-of-the-art transformers. The codes are available at https://github.com/Tangshitao/QuadtreeAttention.

Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints

“…To solve this issue, transformer-based detector-free methods have emerged as more robust alternatives, demonstrating impressive matching abilities in texture-less regions [43,18,47,57,4]. However, the high computational cost of attention limits transformer-based methods to 'semi-dense' matching, where source matching points are spaced apart at intervals of coarse feature space, as shown in Fig.…”

“…Figure 1: QuadTree [47] (a,d) vs our CasMTR (b,c,e). Our method achieves more fine-grained matching pairs for both source and target images (b).…”

“…Figure 2: Illustration of CasMTR pipeline; and our novelties compared against the existing steps from detector-free matching methods [43,47,4] are highlighted in red. without a substantial increase in computational costs.…”

“…Suffixes '-8c', '-4c', and '-2c' denote matching at 1/8, 1/4, and 1/2 of image size. Baseline: QuadTree [47] with the same backbone as ours. Directly implementing QuadTree-4c causes Out-of-memory (OOM) error in a 32GB GPU, so its inference speed is estimated in brackets.…”

Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations

“…We refer to (Fisher, 2012) for an overview of the main variants. Adaptive quadrees have also been successfully introduced in Transformer architectures (Tang et al, 2022), suggesting that further techniques linked to Collages and fractal compression may be beneficial in this domain. Finally, images generated from IFSs have been used to construct artificial pretraining datasets for large vision models (Kataoka et al, 2020).…”