BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection

Li, Yinhao; Zheng, Gengfeng; Yu, Guanyi; Yang, Ji; Wang, Zengran; Shi, Yukang; Sun, Jing; Li, Zeming

doi:10.48550/arxiv.2206.10092

Cited by 15 publications

(97 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among them, BEV grid augmentation is par-ticularly important for this paradigm, which is also mentioned in [62]. In addition, for class-imbalanced issues, similar to LiDAR-based approaches, some methods [62], [82], [129] exploit CBGS [130] to increase the number of samples for long-tailed categories. However, to our best knowledge, there are still very few works targeting this problem.…”

Section: Training Detailsmentioning

confidence: 99%

Vision-Centric BEV Perception: A Survey

Ma¹,

Wang²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

Vision-centric BEV perception has recently received increased attention from both industry and academia due to its inherent merits, including presenting a natural representation of the world and being fusion-friendly. With the rapid development of deep learning, numerous methods have been proposed to address the vision-centric BEV perception. However, there is no recent survey for this novel and growing research field. To stimulate its future research, this paper presents a comprehensive survey of recent progress of vision-centric BEV perception and its extensions. It collects and organizes the recent knowledge, and gives a systematic review and summary of commonly used algorithms. It also provides in-depth analyses and comparative results on several BEV perception tasks, facilitating the comparisons of future works and inspiring future research directions. Moreover, empirical implementation details are also discussed and shown to benefit the development of related algorithms.

show abstract

Section: Training Detailsmentioning

confidence: 99%

Vision-Centric BEV Perception: A Survey

Ma¹,

Wang²,

Bai³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

“…To learn BEV representation from surrounding views, depth-based methods [33,36,21,16,43,35] infer depth in image views and project them to the BEV plane with the extrinsics and intrinsics, where Unsupervised depth estimation remains challenges. Although BEVdepth [21] improves feature extracting and downstream tasks performance compared with other paradigms, additional supervision is a key issue, relatively more difficult to obtain. In contrast, inspired by the transformer framework for 2D object detection [55,6], some works such as BEVformer [22,17,32,8] emphasizes directly learning the transformation relationship between image view and BEV based on the attention mechanism.…”

Section: Related Workmentioning

confidence: 99%

“…Recent approaches emphasize transforming 2D image features to sparse instance-level [9,37,45] or dense Bird's Eye View (BEV) representation [16,22,26], characterizing the 3D structure of the surrounding environment. Although some depth-based detectors [16,17,21,26,51] incorporate depth estimation to introduce such 3D information, the extra depth supervision is acquired for preciser detection. Therefore, other paradigms [22,45] directly learn the transformation based on the attention mechanism [40].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Camera Calibration Free BEV Representation for 3D Object Detection

Jiang¹,

Meng²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

In advanced paradigms of autonomous driving, learning Bird's Eye View (BEV) representation from surrounding views is crucial for multi-task framework. However, existing methods based on depth estimation or camera-driven attention are not stable to obtain transformation under noisy camera parameters, mainly with two challenges, accurate depth prediction and calibration. In this work, we present a completely Multi-Camera Calibration Free Transformer (CFT) for robust BEV representation, which focuses on exploring implicit mapping, not relied on camera intrinsics and extrinsics. To guide better feature learning from image views to BEV, CFT mines potential 3D information in BEV via our designed position-aware enhancement (PA). Instead of camera-driven point-wise or global transformation, for interaction within more effective region and lower computation cost, we propose a view-aware attention which also reduces redundant computation and promotes converge. CFT achieves 49.7% NDS on the nuScenes detection task leaderboard, which is the first work removing camera parameters, comparable to other geometry-guided methods. Without temporal input and other modal information, CFT achieves second highest performance with a smaller image input (1600 × 640). Thanks to view-attention variant, CFT reduces memory and transformer FLOPs for vanilla attention by about 12% and 60%, respectively, with improved NDS by 1.0%. Moreover, its natural robustness to noisy camera parameters makes CFT more competitive.

show abstract

“…Deep neural network-based 3D object detectors, such as those presented in [13,15,18,21,[37][38][39][40]43,44], have demonstrated promising performance on various challenging realworld benchmarks, including the KITTI [9], Waymo [32], and nuScenes [3] datasets. These popular approaches utilize either point clouds [15,37,43,44] or images [13,17,18,21,[38][39][40] as inputs for detection tasks. In comparison to LIDAR-based methods, camera-based approaches have garnered significant attention due to their low deployment cost, high computational efficiency, and dense semantic information.…”

Section: Introductionmentioning

confidence: 99%

On the Adversarial Robustness of Camera-based 3D Object Detection

Xie¹,

Li²,

Wang³

et al. 2023

Preprint

View full text Add to dashboard Cite

In recent years, camera-based 3D object detection has gained widespread attention for its ability to achieve high performance with low computational cost. However, the robustness of these methods to adversarial attacks has not been thoroughly examined. In this study, we conduct the first comprehensive investigation of the robustness of leading camera-based 3D object detection methods under various adversarial conditions. Our experiments reveal five interesting findings: (a) the use of accurate depth estimation effectively improves robustness; (b) depth-estimationfree approaches do not show superior robustness; (c) bird'seye-view-based representations exhibit greater robustness against localization attacks; (d) incorporating multi-frame benign inputs can effectively mitigate adversarial attacks; and (e) addressing long-tail problems can enhance robustness. We hope our work can provide guidance for the design of future camera-based object detection modules with improved adversarial robustness.

show abstract

BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection

Cited by 15 publications

References 0 publications

Vision-Centric BEV Perception: A Survey

Vision-Centric BEV Perception: A Survey

Multi-Camera Calibration Free BEV Representation for 3D Object Detection

On the Adversarial Robustness of Camera-based 3D Object Detection

Contact Info

Product

Resources

About