Anchor DETR: Query Design for Transformer-Based Detector

Abstract

In this paper, we propose a novel query design for the transformer-based object detection. In previous transformer-based detectors, the object queries are a set of learned embeddings. However, each learned embedding does not have an explicit physical meaning and we cannot explain where it will focus on. It is difficult to optimize as the prediction slot of each object query does not have a specific mode. In other words, each object query will not focus on a specific region. To solve these problems, in our query design, object queries are based on anchor points, which are widely used in CNN-based detectors. So each object query focuses on the objects near the anchor point. Moreover, our query design can predict multiple objects at one position to solve the difficulty: ``one region, multiple objects''. In addition, we design an attention variant, which can reduce the memory cost while achieving similar or better performance than the standard attention in DETR. Thanks to the query design and the attention variant, the proposed detector that we called Anchor DETR, can achieve better performance and run faster than the DETR with 10x fewer training epochs. For example, it achieves 44.2 AP with 19 FPS on the MSCOCO dataset when using the ResNet50-DC5 feature for training 50 epochs. Extensive experiments on the MSCOCO benchmark prove the effectiveness of the proposed methods. Code is available at https://github.com/megvii-research/AnchorDETR.

DETRs with Hybrid Matching

“…DETR for object detection. With the pioneering work DETR [5] introducing transformers [57] to 2D object detection, more and more follow-up works [44,13,10,62] have built various advanced extensions based on DETR because it removes the need for many hand-designed components like non-maximum suppression [47] or initial anchor boxes generation [16,51,32,35]. Deformable-DETR [81] introduced the multi-scale deformable self/cross-attention scheme, which attends to only a small set of key sampling points around a reference and achieves better performance than DETR (especially on small objects).…”

“…POTO [59] propose to assign the anchor with either the maximum IoU or closest to the object center as the positive sample, which is modified from the strategies of RetinaNet [32] or FCOS. DETR [5] and its followups [44,4,62,81,34,22] apply the Hungarian matching to compute one-to-one positive assignments based on the global minimum matching cost values between all predictions and the ground-truth boxes. Different from the most related work POTO [59] that only uses one-to-many assignment, based on ATSS [76], to help the classification branch of FCOS [56], our approach chooses Hungarian matching to perform both one-to-one matching and one-to-many matching following DETR and generalizes to various vision tasks.…”

“…Motivated by the success of DETR on a wide variety of vision tasks, many follow-up efforts have improved DETR from various aspects, including redesigning more advanced transformer encoder [81, 13,14] or transformer decoder architectures [44,73,81,4] or query formulations [62,34,22,74]. Different from most of these previous efforts, we focus on the inefficient training issues caused by one-to-one matching, which only assigns one query to each ground truth.…”

Semantic-Aligned Matching for Enhanced DETR Convergence and Multi-Scale Feature Fusion

“…PnP-DETR [54] proposes a poll-and-pool sampling strategy in its attention mechanism. Besides, Conditional DETR [10], SMCA-DETR [11], Anchor DETR [12], and DAB-DETR [55] make substantial modifications to the attention mechanism, aiming to add spatial constraints to the original cross-attention to better focus on prominent regions. Furthermore, the recently proposed DN-DETR [30] designs a novel de-noising training strategy to speed up DETR's training procedure, which also achieves very promising results.…”

“…The object queries after distilling relevant features are used to generate instance-level detection predictions as well as to repeat the subsequent 'matching and feature distillation' processes for refined predictions. However, as pointed out in [9,10,11,12,13], it is difficult for object queries to learn to match appropriate regions. As illustrated in Fig.…”