Sparse and Noisy LiDAR Completion with RGB Guidance and Uncertainty

Abstract

This work proposes a new method to accurately complete sparse LiDAR maps guided by RGB images. For autonomous vehicles and robotics the use of LiDAR is indispensable in order to achieve precise depth predictions. A multitude of applications depend on the awareness of their surroundings, and use depth cues to reason and react accordingly. On the one hand, monocular depth prediction methods fail to generate absolute and precise depth maps. On the other hand, stereoscopic approaches are still significantly outperformed by LiDAR based approaches. The goal of the depth completion task is to generate dense depth predictions from sparse and irregular point clouds which are mapped to a 2D plane. We propose a new framework which extracts both global and local information in order to produce proper depth maps. We argue that simple depth completion does not require a deep network. However, we additionally propose a fusion method with RGB guidance from a monocular camera in order to leverage object information and to correct mistakes in the sparse input. This improves the accuracy significantly. Moreover, confidence masks are exploited in order to take into account the uncertainty in the depth predictions from each modality. This fusion method outperforms the state-of-the-art and ranks first on the KITTI depth completion benchmark [21]. Our code with visualizations is available at https: // github. com/ wvangansbeke/ Sparse-Depth-Completion .

3D LiDAR and Stereo Fusion using Stereo Matching Network with Conditional Cost Volume Normalization

“…Two different settings are explored by several prior works: LiDAR fused with a monocular image or stereo ones. As the depth estimation from a single image is typically based on a regression from pixels, which is inherently unreliable and ambiguous, most of the recent monocular-based works aim to achieve the completion on the sparse depth map obtained by LiDAR sensor with the help of rich information from RGB images [14][15] [16] [17][18] [19], or refine the depth regression by having LiDAR data as a guidance [20] [21].…”

MLOD: A multi-view 3D object detection based on robust feature fusion method

“…But due to the sparsity of the LIDAR point cloud, most of the depth information in the image plane is unknown. Recently, several approaches were proposed to complete the depth map, e.g., [11], [12]. Unfortunately, they typically have high GPU memory usage, and thus are not suitable for our implementation.…”

To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

“…Error Our Approach 892 243 Table 2: Comparison of our depth completion approach against [10,15,16,39,50,54,55] using the validation set in [54]. Despite not being the primary focus, our completion approach remains competitive with the state of the art.…”

“…While our approach is certainly not entirely immune to this issue (Figure 8), it produces visually improved outputs compared to the other techniques, as seen in Figure 5. Numerical results in Table 2 demonstrate that our completion approach quantitatively outperforms many contemporary state-of-the-art comple-tion methods [10,16,50,54] and remains competitive with others [15,39,55], despite the fact that it is primarily incorporated into our pipeline to improve the main functionality of the approach (monocular depth estimation) and lacks the complex training objectives of many of the comparators.…”

“…However, pervasive issues and complications ever-present in depth-reliant vision systems (e.g. missing depth, temporal and in-scene consistency, intensive computational and calibration requirements and alike), have led to the emergence of entire areas of research focusing on refinement procedures post estimation or capture [3,8,10,16,39,55] to render scene depth more useful for downstream applications.…”