DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

Wang, Yue; Guizilini, Vitor; Zhang, Tianyuan; Wang, Yilun; Zhao, Hang; Solomon, Justin

doi:10.48550/arxiv.2110.06922

Cited by 6 publications

(11 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Benefitting from the strong spatial correlation of the targets' attribute with the image appearance, it works well in predicting this but is relatively poor in perceiving the targets' translation, velocity, and depth. Following DETR [3], DETR3D [44] proposes to detect 3D objects in an attention pattern, which achieves similar performance as FCOS3D but with a smaller computing budget and faster inference speed. PGD [43] further develops the FCOS3D paradigm by searching and resolving with the outstanding shortcoming (i.e.…”

Section: Visual Based 3d Object Detectionmentioning

confidence: 99%

“…nuScenes dataset includes 1000 scenes with images from 6 cameras with surrounding views, points from 5 Radars and 1 LiDAR. It is the up-todate popular benchmark for 3D object detection [42,44,43,29] and BEV semantic segmentation [34,30,28,47]. The scenes are officially split into 700/150/150 scenes for training/validation/testing.…”

Section: Experimental Settingsmentioning

confidence: 99%

“…We comprehensively compare the proposed BEVDet with other paradigms like FCOS3D [42], its upgraded version PGD [43], and DETR3D [44]. Their numbers of parameters, computing budget, inference speed, and performance on the nuScenes val set are all listed in Tab.…”

Section: Nuscenes Val Setmentioning

confidence: 99%

See 2 more Smart Citations

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

Huang¹,

Guan²,

Zhu³

et al. 2021

Preprint

168

View full text Add to dashboard Cite

Autonomous driving perceives the surrounding environment for decision making, which is one of the most complicated scenes for visual perception. The great power of paradigm innovation in solving the 2D object detection task inspires us to seek an elegant, feasible, and scalable paradigm for pushing the performance boundary in this area. To this end, we contribute the BEVDet paradigm in this paper. BEVDet is developed by following the principle of detecting the 3D objects in Bird-Eye-View (BEV), where route planning can be handily performed. In this paradigm, four kinds of modules are conducted in succession with different roles: an image-view encoder for encoding feature in image view, a view transformer for feature transformation from image view to BEV, a BEV encoder for further encoding feature in BEV, and a task-specific head for predicting the targets in BEV. We merely reuse the existing modules for constructing BEVDet and make it feasible for multi-camera 3D object detection by constructing an exclusive data augmentation strategy. The proposed paradigm works well in multi-camera 3D object detection and offers a good trade-off between computing budget and performance. BEVDet with 704×256 (1/8 of the competitors) image size scores 29.4% mAP and 38.4% NDS on the nuScenes val set, which is comparable with FCOS3D (i.e., 2008.2 GFLOPs, 1.7 FPS, 29.5% mAP, and 37.2% NDS), while requires just 12% computing budget of 239.4 GFLOPs and runs 4.3 times faster. Scaling up the input size to 1408×512, BEVDet scores 34.9% mAP and 41.7% NDS, which requires just 601.4 GFLOPs and significantly suppresses FCOS3D by 5.4% mAP and 4.5% NDS. The superiority of BEVDet tells the magic of paradigm innovation.

show abstract

Section: Visual Based 3d Object Detectionmentioning

confidence: 99%

Section: Experimental Settingsmentioning

confidence: 99%

See 1 more Smart Citation

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

Huang¹,

Guan²,

Zhu³

et al. 2021

Preprint

168

View full text Add to dashboard Cite

show abstract

“…However, Grassia [15] states that Euler angles and quaternions representation [16] are not ideal to compute and differentiate positions, due to the discontinuity [17] and ambiguity problem. We still observe recent researchers such as [18,19,20,21,22,23,24,25,26,27,28,29,30] using these types of scalar-based angle representation in 3D Object Detection. We are also noticing some variants of the scalar-based methods.…”

Section: -D Scalar Representationsmentioning

confidence: 93%

SoK: Vehicle Orientation Representations for Deep Rotation Estimation

Tu¹,

Peng²,

Leung³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, there is an influx of deep learning models for 3D vehicle object detection. However, little attention was paid to orientation prediction. Existing research work proposed various vehicle orientation representation methods for deep learning, however a holistic, systematic review has not been conducted. Through our experiments, we categorize and compare the accuracy performance of various existing orientation representations using the KITTI 3D object detection dataset [1], and propose a new form of orientation representation: Tricosine. Among these, the 2D Cartesian-based representation [cos(θ), sin(θ)], or Single Bin, achieves the highest accuracy, with additional channeled inputs (positional encoding and depth map) not boosting prediction performance. Our code is published on GitHub: https://github.com/umd-fire-coml/KITTI-orientationlearning

show abstract

“…Transformer-based approaches are another line of research for perception in BEV space. In object detection task, DETR3D [8] introduces a 3D bounding boxes detection method that directly generates predictions in 3D space from 2D features of multiple camera images. The view transformation between 3D space and 2D image space is achieved by 3D-to-2D queries of a cross-attention module.…”

Section: Introductionmentioning

confidence: 99%

BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

Peng¹,

Chen²,

Fu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Semantic segmentation in bird's eye view (BEV) is an important task for autonomous driving. Though this task has attracted a large amount of research efforts, it is still challenging to flexibly cope with arbitrary (single or multiple) camera sensors equipped on the autonomous vehicle. In this paper, we present BEVSegFormer, an effective transformerbased method for BEV semantic segmentation from arbitrary camera rigs. Specifically, our method first encodes image features from arbitrary cameras with a shared backbone. These image features are then enhanced by a deformable transformerbased encoder. Moreover, we introduce a BEV transformer decoder module to parse BEV semantic segmentation results. An efficient multi-camera deformable attention unit is designed to carry out the BEV-to-image view transformation. Finally, the queries are reshaped according the layout of grids in the BEV, and upsampled to produce the semantic segmentation result in a supervised manner. We evaluate the proposed algorithm on the public nuScenes dataset and a self-collected dataset. Experimental results show that our method achieves promising performance on BEV semantic segmentation from arbitrary camera rigs. We also demonstrate the effectiveness of each component via ablation study.

show abstract

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries

Cited by 6 publications

References 36 publications

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

SoK: Vehicle Orientation Representations for Deep Rotation Estimation

BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs

Contact Info

Product

Resources

About