Conditional DETR for Fast Training Convergence

Meng, Depu; Chen, Xiaokang; Fan, Zejia; Zeng, Gang; Li, Houqiang; Yuan, Yuhui; Sun, Lei; Wang, Jingdong

doi:10.48550/arxiv.2108.06152

Cited by 13 publications

(27 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Deformable DETR (Zhu et al, 2021) directly treats 2D reference points as queries and predicts deformable sampling points for each reference point to perform deformable cross-attention operation. Conditional DETR (Meng et al, 2021) decouples the attention formulation and generates positional queries based on reference coordinates. Efficient DETR (Yao et al, 2021) introduces a dense prediction module to select top-K positions as object queries.…”

Section: Related Workmentioning

confidence: 99%

“…For example, SMCA (Gao et al, 2021) speeds up the training by applying pre-defined Gaussian maps around reference points. Conditional DETR (Meng et al, 2021) uses explicit positional embedding as positional queries for training, yielding attention maps similar to Gaussian kernels as shown in Fig. 4…”

Section: Related Workmentioning

confidence: 99%

“…Inspired by Conditional DETR (Meng et al, 2021), we concatenate the position and content information together as queries and keys in the cross-attention module, so that we can decouple the content and position contributions to the query-to-feature similarity computed as the dot product between a query and a key. To rescale the positional embeddings, we leverage the conditional spatial query (Meng et al, 2021) as well. More specifically, we learn a MLP (csq) : R D → R D to obtain a scale vector conditional on the content information and use it perform element-wise multiplication with the positional embeddings:…”

Section: Learning Anchor Boxes Directlymentioning

confidence: 99%

“…However, due to its ineffective design and use of queries, DETR suffers from significantly slow training convergence, usually requiring 500 epochs to achieve a good performance. To address this issue, many follow-up works attempted to improve the design of DETR queries for both faster training convergence and better performance (Zhu et al, 2021;Gao et al, 2021;Meng et al, 2021;. Despite all the progress, the role of the learned queries in DETR is still not fully understood or utilized.…”

Section: Introductionmentioning

confidence: 99%

“…For example, Conditional DETR learns a conditional spatial query by adapting a query based on its content feature for better matching with image features (Meng et al, 2021). Efficient DETR introduces a dense prediction module to select top-K object queries (Yao et al, 2021) and Anchor DETR formulates queries as 2D anchor points , both associating each query with a specific spatial position.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

Liu¹,

Li²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present in this paper a novel query formulation using dynamic anchor boxes for DETR (DEtection TRansformer) and offer a deeper understanding of the role of queries in DETR. This new formulation directly uses box coordinates as queries in Transformer decoders and dynamically updates them layer-by-layer. Using box coordinates not only helps using explicit positional priors to improve the queryto-feature similarity and eliminate the slow training convergence issue in DETR, but also allows us to modulate the positional attention map using the box width and height information. Such a design makes it clear that queries in DETR can be implemented as performing soft ROI pooling layer-by-layer in a cascade manner. As a result, it leads to the best performance on MS-COCO benchmark among the DETR-like detection models under the same setting, e.g., AP 45.7% using ResNet50-DC5 as backbone trained in 50 epochs. We also conducted extensive experiments to confirm our analysis and verify the effectiveness of our methods.Code is available at https://github.com/SlongLiu/DAB-DETR.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Learning Anchor Boxes Directlymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

Liu¹,

Li²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

DN-DETR: Accelerate DETR Training by Introducing Query DeNoising

Li,

Zhang,

Liu

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We present in this paper a novel denoising training method to speed up DETR (DEtection TRansformer) training and offer a deepened understanding of the slow convergence issue of DETR-like methods. We show that the slow convergence results from the instability of bipartite graph matching which causes inconsistent optimization goals in early training stages. To address this issue, except for the Hungarian loss, our method additionally feeds GT bounding boxes with noises into the Transformer decoder and trains the model to reconstruct the original boxes, which effectively reduces the bipartite graph matching difficulty and leads to faster convergence. Our method is universal and can be easily plugged into any DETR-like method by adding dozens of lines of code to achieve a remarkable improvement. As a result, our DN-DETR results in a remarkable improvement (+1.9AP) under the same setting and achieves 46.0 AP and 49.5 AP trained for 12 and 50 epochs with the ResNet-50 backbone. Compared with the baseline under the same setting, DN-DETR achieves comparable performance with 50% training epochs. We also demonstrate the effectiveness of denoising training in CNN-based detectors (Faster R-CNN), segmentation models (Mask2Former, Mask DINO), and more DETR-based models (DETR, Anchor DETR, Deformable DETR). Code is available at https://github.com/IDEA-Research/DN-DETR.

show abstract

Anchor DETR: Query Design for Transformer-Based Detector

Wang

Zhang

Yang

et al. 2022

AAAI

172

View full text Add to dashboard Cite

In this paper, we propose a novel query design for the transformer-based object detection. In previous transformer-based detectors, the object queries are a set of learned embeddings. However, each learned embedding does not have an explicit physical meaning and we cannot explain where it will focus on. It is difficult to optimize as the prediction slot of each object query does not have a specific mode. In other words, each object query will not focus on a specific region. To solve these problems, in our query design, object queries are based on anchor points, which are widely used in CNN-based detectors. So each object query focuses on the objects near the anchor point. Moreover, our query design can predict multiple objects at one position to solve the difficulty: ``one region, multiple objects''. In addition, we design an attention variant, which can reduce the memory cost while achieving similar or better performance than the standard attention in DETR. Thanks to the query design and the attention variant, the proposed detector that we called Anchor DETR, can achieve better performance and run faster than the DETR with 10x fewer training epochs. For example, it achieves 44.2 AP with 19 FPS on the MSCOCO dataset when using the ResNet50-DC5 feature for training 50 epochs. Extensive experiments on the MSCOCO benchmark prove the effectiveness of the proposed methods. Code is available at https://github.com/megvii-research/AnchorDETR.

show abstract

Conditional DETR for Fast Training Convergence

Cited by 13 publications

References 0 publications

DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR

DN-DETR: Accelerate DETR Training by Introducing Query DeNoising

Anchor DETR: Query Design for Transformer-Based Detector

Contact Info

Product

Resources

About