Non-Local Spatial Propagation Network for Depth Completion

Park, Jinsun; Joo, Kyungdon; Hu, Zhe; Liu, Chi-Kuei; Kweon, In So

doi:10.48550/arxiv.2007.10042

Cited by 13 publications

(31 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our method even can outperform some early-stage supervised solutions. Recent developed supervised models are prone to use a large network with 10 million or even 100 million parameters [41], [25], [42], thus have better performance. Besides, we also list two state-of-the-art supervised methods in Table III.…”

Section: The Performance Of Surface Geometry Modelmentioning

confidence: 99%

A Surface Geometry Model for LiDAR Depth Completion

Zhao

Bai

Zhang

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

LiDAR depth completion is a task that predicts depth values for every pixel on the corresponding camera frame although only sparse LiDAR points are available. Most of the existing state-of-the-art solutions are based on deep neural networks, which need a large amount of data and heavy computations for training the models. In this letter, a novel nonlearning depth completion method is proposed by exploiting the local surface geometry that is enhanced by an outlier removal algorithm. The proposed surface geometry model is inspired by the observation that most pixels with unknown depth have a nearby LiDAR point. Therefore, it is assumed those pixels share the same surface with the nearest LiDAR point, and their respective depth can be estimated as the nearest LiDAR depth value plus a residual error. The residual error is calculated by using a derived equation with several physical parameters as input, including the known camera intrinsic parameters, estimated normal vector, and offset distance on the image plane. The proposed method is further enhanced by an outlier removal algorithm that is designed to remove incorrectly mapped LiDAR points from occluded regions. On KITTI dataset, the proposed solution achieves the best error performance among all existing non-learning methods and is comparable to the best self-supervised learning method and some supervised learning methods. Moreover, since removing outlier points coming from occluded regions is a commonly existing problem in LiDAR-Camera systems, the proposed outlier removal algorithm is a general preprocessing step that is applicable to many robotic systems with both camera and LiDAR sensors.

show abstract

Section: The Performance Of Surface Geometry Modelmentioning

confidence: 99%

A Surface Geometry Model for LiDAR Depth Completion

Zhao

Bai

Zhang

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…Depth Completion: The depth completion is a task to generate per-pixel dense depth maps from the relative sparse depths. Most depth completion models [23,24,25,26,27,28] are trained with labeled data which require intensive human labors. To utilize the massive unlabeled data, self-supervised depth completion methods were developed in recent years [29,30,31,32,33] to generate depth maps from 64-beams dense LiDAR points.…”

Section: Related Workmentioning

confidence: 99%

Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR

Feng¹,

Jing²,

Yin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-supervised monocular depth prediction provides a cost-effective solution to obtain the 3D location of each pixel. However, the existing approaches usually lead to unsatisfactory accuracy, which is critical for autonomous robots. In this paper, we propose a novel two-stage network to advance the self-supervised monocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) Li-DAR. Unlike the existing methods that use sparse LiDAR mainly in a manner of time-consuming iterative post-processing, our model fuses monocular image features and sparse LiDAR features to predict initial depth maps. Then, an efficient feed-forward refine network is further designed to correct the errors in these initial depth maps in pseudo-3D space with real-time performance. Extensive experiments show that our proposed model significantly outperforms all the stateof-the-art self-supervised methods, as well as the sparse-LiDAR-based methods on both self-supervised monocular depth prediction and completion tasks. With the accurate dense depth prediction, our model outperforms the state-of-the-art sparse-LiDAR-based method (Pseudo-LiDAR++ [1]) by more than 68% for the downstream task monocular 3D object detection on the KITTI Leaderboard.

show abstract

“…The term depth completion is used when the input is RGBD, where the D (depth) channel is noisy and may have missing values. Existing methods for single-view depth estimation [1,4,9,10,18,19,24,29,30,40] and depth completion [15,25,27,31,42] improve depth prediction for the entire image, relying on reconstructed 3D mesh data that is assumed to provide accurate depth. Chabra et al [5] show that an exclusion mask for noisy areas such as reflective surfaces can result in better reconstruction.…”

Section: D Plane Detection and Plane Reconstructionmentioning

confidence: 99%

Mirror3D: Depth Refinement for Mirror Surfaces

Tan

Lin

Chang

et al. 2021

Preprint

View full text Add to dashboard Cite

Input image + mirror mask Raw depth Original point cloud Refined depth Refined point cloud Figure 1: We present the task of 3D mirror plane prediction and depth refinement. First, we annotate several popular RGBD datasets (Matterport3D [6], ScanNet [7], NYUv2 [32]) with 3D mirror planes. Our benchmarks show that both existing RGBD dataset 'ground truth' raw depth data, and state-of-the-art depth estimation and depth completion methods exhibit dramatic errors on mirror surfaces. We propose an architecture for 3D mirror plane estimation that refines depth estimates and produces more reliable reconstructions (compare left and right depth and point cloud pairs from NYUv2 [32] dataset).

show abstract

Non-Local Spatial Propagation Network for Depth Completion

Cited by 13 publications

References 21 publications

A Surface Geometry Model for LiDAR Depth Completion

A Surface Geometry Model for LiDAR Depth Completion

Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR

Mirror3D: Depth Refinement for Mirror Surfaces

Contact Info

Product

Resources

About