Multi-Task Multi-Sensor Fusion for 3D Object Detection

Liang, Ming; Yang, Bin; Chen, Yun; Hu, Rui; Urtasun, Raquel

doi:10.1109/cvpr.2019.00752

Cited by 588 publications

(339 citation statements)

References 29 publications

Supporting

Mentioning

338

Contrasting

Order By: Relevance

“…Autonomous Driving Datasets with Trajectory Data. ApolloScape [26] also uses sensor-equipped vehicles to observe driving trajectories in the wild and presents a forecasting benchmark [41] [37] show how 3D object detection accuracy can be improved by using mapping (ground height estimation) as an additional task in multi-task learning. Suraj et al [40] use dashboardmounted monocular cameras on a fleet of vehicles to build a 3D map via city-scale structure-from-motion for localization of ego-vehicles and trajectory extraction.…”

Section: Related Workmentioning

confidence: 99%

Argoverse: 3D Tracking and Forecasting With Rich Maps

Chang

Ramanan²,

Hays³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

930

739

View full text Add to dashboard Cite

Figure 1: We introduce datasets for 3D tracking and motion forecasting with rich maps for autonomous driving. Our 3D tracking dataset contains sequences of LiDAR measurements, 360 • RGB video, front-facing stereo (middle-right), and 6-dof localization. All sequences are aligned with maps containing lane center lines (magenta), driveable region (orange), and ground height. Sequences are annotated with 3D cuboid tracks (green). A wider map view is shown in the bottom-right. AbstractWe present Argoverse -two datasets designed to support autonomous vehicle machine learning tasks such as 3D tracking and motion forecasting. Argoverse was collected by a fleet of autonomous vehicles in Pittsburgh and Miami. The Argoverse 3D Tracking dataset includes 360 • images from 7 cameras with overlapping fields of view, 3D point clouds from long range LiDAR, 6-DOF pose, and 3D track annotations. Notably, it is the only modern AV dataset that provides forward-facing stereo imagery. The Argoverse Motion Forecasting dataset includes more than 300,000 5 second tracked scenarios with a particular vehicle identified for trajectory forecasting. Argoverse is the first autonomous vehicle dataset to include "HD maps" with 290 km of mapped lanes with geometric and semantic metadata. All data is released under a Creative Commons license at *Equal contribution www.argoverse.org. In our baseline experiments, we illustrate how detailed map information such as lane direction, driveable area, and ground height improves the accuracy of 3D object tracking and motion forecasting. Our tracking and forecasting experiments represent only an initial exploration of the use of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

show abstract

Section: Related Workmentioning

confidence: 99%

Argoverse: 3D Tracking and Forecasting With Rich Maps

Chang

Ramanan²,

Hays³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

930

739

View full text Add to dashboard Cite

show abstract

“…F-PointNet [20] extends the 2D detections from image into corresponding frustums in the 3D space. MMF [13] is proposed to exploit multiple related tasks including depth completion and 2D object detection for accurate multi-sensor 3D object detection. However, although multiple sensors could provide extra information, the inference efficiency for these frameworks are relatively low.…”

Section: D Object Detectionmentioning

confidence: 99%

“…Multi-scale strategy [14,13,2] has been proved to be effective to 3D object detection. ContFuse [14] uses continuous convolution to aggregate multi-scale feature maps from different ResNet Block [7].…”

Section: Multi-scale Feature Aggregationmentioning

confidence: 99%

“…Methods should be at least 20Hz since onboard application should cover 360 degree rather than KITTI annotation at limited 90 degree. Drawn methods are FP: F-PointNet [20], AF: AVOD-FPN [9], M: MMF [13], I: IPOD [31], FC: F-ConvNet [26], S: STD [32], PR: PointRCNN [22], FPR: Fast Point R-CNN [2], SE: SECOND [28], PP: PointPillars [10], PI: PIXOR++ [29] and O: our HVNet. For PointPillars we use their runtime on PyTorch for a fair comparison.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SARPNET: Shape attention regional proposal network for liDAR-based 3D object detection

Chen

Zhang

et al. 2020

Neurocomputing

View full text Add to dashboard Cite

“…Apart from [84], [155], the methods which directly detect 3D BBs use stereo images [92], RGB-D cameras [86], [93], [94], [95], [152], and LIDAR sensors [152] as input. There are also several methods fusing multiple sensor inputs, and generating 3D BB hypotheses [107]. Without depending on the input (RGB (Mo/St), RGB-D, or LIDAR), 3D BB detection methods produce oriented 3D BBs, which are parameterized with center x = (x, y, z), size d = (d w , d h , d l ), and orientation (θ y ) around the gravity direction.…”

Section: Introductionmentioning

confidence: 99%

A review on object pose recovery: From 3D bounding box detectors to full 6D pose estimators

Şahin

Garcia-Hernando

Sock

et al. 2020

Image and Vision Computing

View full text Add to dashboard Cite

Object pose recovery has gained increasing attention in the computer vision field as it has become an important problem in rapidly evolving technological areas related to autonomous driving, robotics, and augmented reality. Existing review-related studies have addressed the problem at visual level in 2D, going through the methods which produce 2D bounding boxes of objects of interest in RGB images. The 2D search space is enlarged either using the geometry information available in the 3D space along with RGB (Mono/Stereo) images, or utilizing depth data from LIDAR sensors and/or RGB-D cameras. 3D bounding box detectors, producing category-level amodal 3D bounding boxes, are evaluated on gravity aligned images, while full 6D object pose estimators are mostly tested at instance-level on the images where the alignment constraint is removed. Recently, 6D object pose estimation is tackled at the level of categories. In this paper, we present the first comprehensive and most recent review of the methods on object pose recovery, from 3D bounding box detectors to full 6D pose estimators. The methods mathematically model the problem as a classification, regression, classification & regression, template matching, and point-pair feature matching task. Based on this, a mathematical-model-based categorization of the methods is established. Datasets used for evaluating the methods are investigated with respect to the challenges, and evaluation metrics are studied. Quantitative results of experiments in the literature are analysed to show which category of methods best performs across what types of challenges. The analyses are further extended comparing two methods, which are our own implementations, so that the outcomes from the public results are further solidified. Current position of the field is summarized regarding object pose recovery, and possible research directions are identified.

show abstract

Multi-Task Multi-Sensor Fusion for 3D Object Detection

Cited by 588 publications

References 29 publications

Argoverse: 3D Tracking and Forecasting With Rich Maps

Argoverse: 3D Tracking and Forecasting With Rich Maps

SARPNET: Shape attention regional proposal network for liDAR-based 3D object detection

A review on object pose recovery: From 3D bounding box detectors to full 6D pose estimators

Contact Info

Product

Resources

About