Categorical Depth Distribution Network for Monocular 3D Object Detection

Reading, Cody; Harakeh, Ali; Chae, Julia; Waslander, Steven L.

doi:10.1109/cvpr46437.2021.00845

Cited by 329 publications

(196 citation statements)

References 31 publications

Supporting

Mentioning

154

Contrasting

Order By: Relevance

“…Another solution is to transform image features into BEV features and predict 3D bounding boxes from the top-down view. Methods transform image features into BEV features with the depth information from depth estimation [46] or categorical depth distribution [34]. OFT [36] and ImVoxelNet [37] project the predefined voxels onto image features to generate the voxel representation of the scene.…”

Section: Camera-based 3d Perceptionmentioning

confidence: 99%

“…The bird's-eye-view (BEV) is a commonly used representation of the surrounding scene since it clearly presents the location and scale of objects and is suitable for various autonomous driving tasks, such as perception and planning [29]. Although previous map segmentation methods demonstrate BEV's effectiveness [32,18,29], BEV-based approaches have not shown significant advantages over other paradigm in 3D object detections [47,31,34]. The underlying reason is that the 3D object detection task requires strong BEV features to support accurate 3D bounding box prediction, but generating BEV from the 2D planes is ill-posed.…”

Section: Introductionmentioning

confidence: 99%

“…The underlying reason is that the 3D object detection task requires strong BEV features to support accurate 3D bounding box prediction, but generating BEV from the 2D planes is ill-posed. A popular BEV framework that generates BEV features is based on depth information [46,32,34], but this paradigm is sensitive to the accuracy of depth values or the depth distributions. The detection performance of BEV-based methods is thus subject to compounding errors [47], and inaccurate BEV features can seriously hurt the final performance.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Li¹,

Wang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

3D visual perception tasks, including 3D detection and map segmentation based on multi-camera images, are essential for autonomous driving systems. In this work, we present a new framework termed BEVFormer, which learns unified BEV representations with spatiotemporal transformers to support multiple autonomous driving perception tasks. In a nutshell, BEVFormer exploits both spatial and temporal information by interacting with spatial and temporal space through predefined grid-shaped BEV queries. To aggregate spatial information, we design a spatial cross-attention that each BEV query extracts the spatial features from the regions of interest across camera views. For temporal information, we propose a temporal self-attention to recurrently fuse the history BEV information. Our approach achieves the new state-of-the-art 56.9% in terms of NDS metric on the nuScenes test set, which is 9.0 points higher than previous best arts and on par with the performance of LiDAR-based baselines. We further show that BEVFormer remarkably improves the accuracy of velocity estimation and recall of objects under low visibility conditions. The code will be released at https://github.com/zhiqi-li/BEVFormer.

show abstract

Section: Camera-based 3d Perceptionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Li¹,

Wang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…MonoFENet [1] estimates disparity from the input monocular image, the features of both the 2D and 3D streams can be enhanced and utilized for accurate 3D localization. CaDDN [34] uses a predicted categorical depth distribution for each pixel to project rich contextual feature information to the appropriate depth interval in 3D space then get the final result. M3DSSD [26] proposes a two-step feature alignment approach to overcome feature mismatching.…”

Section: Monocular 3d Object Detectionmentioning

confidence: 99%

Shape Prior Non-Uniform Sampling Guided Real-time Stereo 3D Object Detection

Gao,

Cao,

Pang

2021

Preprint

View full text Add to dashboard Cite

Pseudo-LiDAR based 3D object detectors have gained popularity due to their high accuracy. However, these methods need dense depth supervision and suffer from inferior speed. To solve these two issues, a recently introduced RTS3D builds an efficient 4D Feature-Consistency Embedding (FCE) space for the intermediate representation of object without depth supervision. FCE space splits the entire object region into 3D uniform grid latent space for feature sampling point generation, which ignores the importance of different object regions. However, we argue that, compared with the inner region, the outer region plays a more important role for accurate 3D detection. To encode more information from the outer region, we propose a shape prior non-uniform sampling strategy that performs dense sampling in outer region and sparse sampling in inner region. As a result, more points are sampled from the outer region and more useful features are extracted for 3D detection. Further, to enhance the feature discrimination of each sampling point, we propose a high-level semantic enhanced FCE module to exploit more contextual information and suppress noise better. Experiments on the KITTI dataset are performed to show the effectiveness of the proposed method. Compared with the baseline RTS3D, our proposed method has 2.57% improvement on AP 3d almost without extra network parameters. Moreover, our proposed method outperforms the state-of-the-art methods without extra supervision at a real-time speed.

show abstract

“…Such monocular methods lack the ability of localizing features in a frustum due to depth ambiguity. CaDDN [24] copes with this problem by estimating depth distribution for each pixel before projecting the features to 3D. Similarly for stereo, DSGN [4] builds a plane sweep volume (PSV) distributed using stereo image features to estimate per pixel depth as an auxiliary task in addition to 3D object detection.…”

Section: Monocular and Stereo Detectionmentioning

confidence: 99%

Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles

Sidhu¹,

Wang²,

Naseer³

et al. 2021

Preprint

View full text Add to dashboard Cite

In autonomous driving, there has been an explosion in the use of deep neural networks for perception, prediction and planning tasks. As autonomous vehicles (AVs) move closer to production, multi-modal sensor inputs and heterogeneous vehicle fleets with different sets of sensor platforms are becoming increasingly common in the industry. However, neural network architectures typically target specific sensor platforms and are not robust to changes in input, making the problem of scaling and model deployment particularly difficult. Furthermore, most players still treat the problem of optimizing software and hardware as entirely independent problems. We propose a new end to end architecture, Generalized Sensor Fusion (GSF), which is designed in such a way that both sensor inputs and target tasks are modular and modifiable. This enables AV system designers to easily experiment with different sensor configurations and methods and opens up the ability to deploy on heterogeneous fleets using the same models that are shared across a large engineering organization. Using this system, we report experimental results where we demonstrate near-parity of an expensive high-density (HD) LiDAR sensor with a cheap low-density (LD) LiDAR plus camera setup in the 3D object detection task. This paves the way for the industry to jointly design hardware and software architectures as well as large fleets with heterogeneous configurations.

show abstract

Categorical Depth Distribution Network for Monocular 3D Object Detection

Cited by 329 publications

References 31 publications

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers

Shape Prior Non-Uniform Sampling Guided Real-time Stereo 3D Object Detection

Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles

Contact Info

Product

Resources

About