RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving

Li, Peixuan; Zhao, Huaici; Liu, Pengfei; Cao, Feidao

doi:10.48550/arxiv.2001.03343

Cited by 32 publications

(49 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A quantitative comparison between 2D and 3D detectors is included in Section 5.2. methods. On state-of-the-art 3D detection benchmarks [2,12], state-of-the-art monocular methods [26,41] achieve about half the mAP detection accuracy, of standard Lidar based baselines [60]. Pseudo-Lidar [56] based methods produce a virtual point cloud from RGB images, similar to our approach.…”

Section: Related Workmentioning

confidence: 73%

Multimodal Virtual Point 3D Detection

Yin¹

2021

Preprint

View full text Add to dashboard Cite

Lidar-based sensing drives current autonomous vehicles. Despite rapid progress, current Lidar sensors still lag two decades behind traditional color cameras in terms of resolution and cost. For autonomous driving, this means that large objects close to the sensors are easily visible, but far-away or small objects comprise only one measurement or two. This is an issue, especially when these objects turn out to be driving hazards. On the other hand, these same objects are clearly visible in onboard RGB sensors. In this work, we present an approach to seamlessly fuse RGB sensors into Lidar-based 3D recognition. Our approach takes a set of 2D detections to generate dense 3D virtual points to augment an otherwise sparse 3D point cloud. These virtual points naturally integrate into any standard Lidar-based 3D detectors along with regular Lidar measurements. The resulting multi-modal detector is simple and effective. Experimental results on the large-scale nuScenes dataset show that our framework improves a strong CenterPoint baseline by a significant 6.6 mAP, and outperforms competing fusion approaches. Code and more visualizations are available at https://tianweiy.github.io/mvp/.

show abstract

Section: Related Workmentioning

confidence: 73%

Multimodal Virtual Point 3D Detection

Yin¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Such methods are useful for traffic studies that require counts of vehicle turns and traffic queues, and not necessarily full 3D bounding boxes to represent moving objects. While the sensors' mounting is not the same, methods for 3D object detection in the context of autonomous driving have developed interesting ideas such as carefully designed loss functions [22], the use of auxiliary CAD models [23], geometric constraints between 2D and 3D bounding boxes [24], stereo imaging [25], and keypoint detection [26] [27]. However, most of these methods are designed for 3D detection up to 40m [25][28] and their direct implementation at a longer range produces significantly less accurate results, especially when looking at roads with changing slope.…”

Section: B Image-based 3d Object Detectionmentioning

confidence: 99%

UrbanNet: Leveraging Urban Maps for Long Range 3D Object Detection

Carrillo¹,

Waslander²

2021

Preprint

View full text Add to dashboard Cite

Relying on monocular image data for precise 3D object detection remains an open problem, whose solution has broad implications for cost-sensitive applications such as traffic monitoring. We present UrbanNet, a modular architecture for long range monocular 3D object detection with static cameras. Our proposed system combines commonly available urban maps along with a mature 2D object detector and an efficient 3D object descriptor to accomplish accurate detection at long range even when objects are rotated along any of their three axes. We evaluate UrbanNet on a novel challenging synthetic dataset and highlight the advantages of its design for traffic detection in roads with changing slope, where the flat ground approximation does not hold. Data and code are available at https://github.com/TRAILab/UrbanNet

show abstract

“…There are mainly two types of methods: RGB imagebased and pseudo-LiDAR based. Former employ detection networks like CenterNet [34] to predict the bounding boxes [35,36,37,38] directly from images. The latter perform the detection over the pseudo-LiDAR representation, which is lifted from the dense depth prediction [39,40,41,42].…”

Section: Scale-invariant Lossmentioning

confidence: 99%

Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR

Feng¹,

Jing²,

Yin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-supervised monocular depth prediction provides a cost-effective solution to obtain the 3D location of each pixel. However, the existing approaches usually lead to unsatisfactory accuracy, which is critical for autonomous robots. In this paper, we propose a novel two-stage network to advance the self-supervised monocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) Li-DAR. Unlike the existing methods that use sparse LiDAR mainly in a manner of time-consuming iterative post-processing, our model fuses monocular image features and sparse LiDAR features to predict initial depth maps. Then, an efficient feed-forward refine network is further designed to correct the errors in these initial depth maps in pseudo-3D space with real-time performance. Extensive experiments show that our proposed model significantly outperforms all the stateof-the-art self-supervised methods, as well as the sparse-LiDAR-based methods on both self-supervised monocular depth prediction and completion tasks. With the accurate dense depth prediction, our model outperforms the state-of-the-art sparse-LiDAR-based method (Pseudo-LiDAR++ [1]) by more than 68% for the downstream task monocular 3D object detection on the KITTI Leaderboard.

show abstract

RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving

Cited by 32 publications

References 47 publications

Multimodal Virtual Point 3D Detection

Multimodal Virtual Point 3D Detection

UrbanNet: Leveraging Urban Maps for Long Range 3D Object Detection

Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR

Contact Info

Product

Resources

About