Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud

Abstract

Monocular 3D scene understanding tasks, such as object size estimation, heading angle estimation and 3D localization, is challenging. Successful modern day methods for 3D scene understanding require the use of a 3D sensor. On the other hand, single image based methods have significantly worse performance. In this work, we aim at bridging the performance gap between 3D sensing and 2D sensing for 3D object detection by enhancing LiDAR-based algorithms to work with single image input. Specifically, we perform monocular depth estimation and lift the input image to a point cloud representation, which we call pseudo-LiDAR point cloud. Then we can train a LiDAR-based 3D detection network with our pseudo-LiDAR end-to-end. Following the pipeline of two-stage 3D detection algorithms, we detect 2D object proposals in the input image and extract a point cloud frustum from the pseudo-LiDAR for each proposal. Then an oriented 3D bounding box is detected for each frustum. To handle the large amount of noise in the pseudo-LiDAR, we propose two innovations: (1) use a 2D-3D bounding box consistency constraint, adjusting the predicted 3D bounding box to have a high overlap with its corresponding 2D proposal after projecting onto the image; (2) use the instance mask instead of the bounding box as the representation of 2D proposals, in order to reduce the number of points not belonging to the object in the point cloud frustum. Through our evaluation on the KITTI benchmark, we achieve the top-ranked performance on both bird's eye view and 3D object detection among all monocular methods, effectively quadrupling the performance over previous state-of-the-art. Our code is available at https: //github.com/xinshuoweng/Mono3D_PLiDAR.

Visual-LiDAR Based 3D Object Detection and Tracking for Embedded Systems

“…To evaluate the true potential and fair comparison with state-of-the-art systems, MODT is evaluated on the KITTI datasets with ground truths [44], based on the MOT16 evaluation metrics proposed in [41]. Whereas, the MOT component is independently evaluated using off-the-shelf 3D object detector [45], [46] against the 3D MOT evaluation extension [19].…”

Object spatial localization by fusing 3D point clouds and instance segmentation

“…Table 2 shows the localization results at a short distance, Table 3 shows the localization results at a medium distance, and Table 4 shows the localization results at a long distance. We use the formula (1) to calculate the location accuracy, depth p represents the predicted depth and depth t represents the true depth, "|*|" represents the absolute value calculation. In order to get the effective accuracy, that is, the object predicted depth is comparable with the real depth, the predicted depth should not be more than twice the real depth, otherwise it is meaningless.…”

“…Hardware-assisted methods rely on non-visual sensors. Weng et al proposed a monocular three-dimensional object detection algorithm based on pseudo-laser point cloud [1], Yang et al proposed and implemented a monocular vision SLAM-based UAV autonomous landing system with fusion optical equipment and image recognition technology [2], Wei et al proposed a monocular image obstacle detection algorithm that combines laser radar sparse point clouds and camera natural image density clustering [3], mainly focused on obstacle detection, indoor localization, indoor path planning and other application scenarios [4][5][6][7][8]. Traditional geometric methods are generally based on binocular vision.…”

Forecasting Time-to-Collision from Monocular Video: Feasibility, Dataset, and Challenges

“…However, in order to predict future trajectory reliably, these methods rely on accurate human trajectory history. This usually involves multi-people detection [23], [24], [25], [26] and tracking, and thus has two major disadvantages: (1) Data association is very challenging in crowded scenarios. Small mistakes can be misinterpreted to be very large changes of velocity, resulting in very bad trajectory estimate; (2) Robust multi-person tracking from mobile platform is often timeconsuming.…”