HMS-Net: Hierarchical Multi-scale Sparsity-invariant Network for Sparse Depth Completion

Huang, Zixuan; Fan, Junming; Cheng, Shenggan; Yi, Shuai; Wang, Xiaogang; Li, Hongsheng

doi:10.48550/arxiv.1808.08685

Cited by 10 publications

(19 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although we mainly focus on the outdoor application scenarios, we also train our model on indoor scenes, i.e., NYU-Depth-v2 dataset. As NYU-Depth-v2 dataset provides relatively denser depth measurements by Microsoft Kinect, we uniformly sample the depth map to obtain the sparser version following previous works [26,14]. We compare our results with latest CNN-based methods [26,2] as well as the classic methods [21,35,20] as shown in Table 3, and our method achieves state-of-the-art performance as well.…”

Section: Analysis Of Generalization Ability and Stabilitymentioning

confidence: 95%

“…Ma et al concatenated the sparse depth and color image as the inputs of an off-the-shelf network [26] and further explored the feasibility of self-supervised Li-DAR completion [23]. Moreover, [14,16,33,4] proposed different network architectures to better exploit the potential of the encoder-decoder framework. However, the encoderdecoder architecture tends to predict the depth maps comprehensively but fails to concentrate on the local areas.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Zhu

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

218

142

View full text Add to dashboard Cite

Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse Li-DAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model. AbstractSorry about this little trick and hope it would work, since I do not have much time to neatten my original source code. depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a

show abstract

Section: Analysis Of Generalization Ability and Stabilitymentioning

confidence: 95%

Section: Related Workmentioning

confidence: 99%

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Zhu

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

218

142

View full text Add to dashboard Cite

show abstract

“…Instead of simulating a LiDAR via ray-casting, which is computationally expensive and hard to implement [47], we leverage the z-buffer of our synthetic rendering engine to provide a dense depth ground truth at first. Previous approaches used synthetic sparse data to evaluate a model in indoor scenes or synthetic outdoor scenes [6,35,48]. To sparsify the data a Bernoulli distribution per pixel is used in some works [2,35,48] which, given a probability p B and a dense depth image x D , samples each of the pixels x D,k by either keeping the value x D,k with probability p B or setting its value to 0 with probability (1 − p B ), thus generating the sparse depth x s B D .…”

Section: Data Generation Via Projectionsmentioning

confidence: 99%

“…Previous approaches used synthetic sparse data to evaluate a model in indoor scenes or synthetic outdoor scenes [6,35,48]. To sparsify the data a Bernoulli distribution per pixel is used in some works [2,35,48] which, given a probability p B and a dense depth image x D , samples each of the pixels x D,k by either keeping the value x D,k with probability p B or setting its value to 0 with probability (1 − p B ), thus generating the sparse depth x s B D . We argue that using x s B D does not simulate well a real LiDAR input, thus a model trained with x s B D does not perform well in the real domain.…”

Section: Data Generation Via Projectionsmentioning

confidence: 99%

Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data

Lopez-Rodriguez¹,

Busam²,

Mikolajczyk³

2020

Preprint

View full text Add to dashboard Cite

Depth completion aims to predict a dense depth map from a sparse depth input. The acquisition of dense ground truth annotations for depth completion settings can be difficult and, at the same time, a significant domain gap between real LiDAR measurements and synthetic data has prevented from successful training of models in virtual settings. We propose a domain adaptation approach for sparse-to-dense depth completion that is trained from synthetic data, without annotations in the real domain or additional sensors. Our approach simulates the real sensor noise in an RGB + LiDAR set-up, and consists of three modules: simulating the real LiDAR input in the synthetic domain via projections, filtering the real noisy LiDAR for supervision and adapting the synthetic RGB image using a CycleGAN [1] approach. We extensively evaluate these modules against the state-of-the-art in the KITTI depth completion benchmark, showing significant improvements.

show abstract

“…Konno et al [39]used residual interpolation to combine low-resolution depth maps with high-resolution RGB images to generate high-resolution depth maps. Based on [9], Huang et al [40]utilized the sparsity-invariant layer to design a sparsity-invariant multi-scale encoder-decoder network for sparse depth completion with RGB guidance. Zhang et al [41]proposed to predict surface normal and occlusion boundary from a deep network and further utilize them to help depth completion in indoor scenes.…”

Section: Guided Depth Upsamplingmentioning

confidence: 99%

Depth Edge Guided CNNs for Sparse Depth Upsampling

Guo,

Liu

2020

Preprint

View full text Add to dashboard Cite

Guided sparse depth upsampling aims to upsample an irregularly sampled sparse depth map when an aligned high-resolution color image is given as guidance. Many neural networks have been designed for this task. However, they often ignore the structural difference between the depth and the color image, resulting in obvious artifacts such as texture copy and depth blur at the upsampling depth. Inspired by the normalized convolution operation, we propose a guided convolutional layer to recover dense depth from sparse and irregular depth image with an depth edge image as guidance. Our novel guided network can prevent the depth value from crossing the depth edge to facilitate upsampling. We further design a convolution network based on proposed convolutional layer to combine the advantages of different algorithms and achieve better performance. We conduct comprehensive experiments to verify our method on real-world indoor and synthetic outdoor datasets. Our method produces strong results. It outperforms state-of-the-art methods on the Virtual KITTI dataset and the Middlebury dataset. It also presents strong generalization capability under different 3D point densities, various lighting and weather conditions.

show abstract

HMS-Net: Hierarchical Multi-scale Sparsity-invariant Network for Sparse Depth Completion

Cited by 10 publications

References 40 publications

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data

Depth Edge Guided CNNs for Sparse Depth Upsampling

Contact Info

Product

Resources

About