Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation

Jaritz, Maximilian; Charette, Raoul de; Wirbel, Émilie; Perrotton, Xavier; Nashashibi, Fawzi

doi:10.48550/arxiv.1808.00769

Cited by 7 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ma et al [8] used an early fusion scheme to combine sparse depth input with the corresponding RGB image, which was demonstrated to perform very well. On the other hand, Jaritz et al [7] argued that late-fusion performs better with their proposed architecture, which was also demonstrated in [5]. Wirges et al [15] used a combination of RGB images and surface normals to guide the process of depth upsampling.…”

Section: Related Workmentioning

confidence: 96%

“…Hua and Gong [5] proposed a similar layer, which uses the trained convolution filter to normalize the sparse input. Contrarily, Jaritz et al [7] compared different architectures and argued that the use of validity masks degrades the performance due to the saturation of the masks at early layers within the CNN. This effect is avoided by the use of continuous confidences as proposed in our prior work on unguided depth completion [11].…”

Section: Related Workmentioning

confidence: 99%

“…Validity masks have been extensively used in the literature [1], [2], [3], [4], [5], [6] to inform the learning method about the missing regions in the data. However, validity masks suffer from saturation in multi-stage learning such as Convolutional Neural Networks (CNNs) as shown by [7]. Instead, the saturation problem can be avoided by treating the binary validity masks as continuous confidence fields describing the reliability of the data.…”

Section: Introductionmentioning

confidence: 99%

“…Most recent works for solving the scene depth completion task are based on CNNs, and show a great success [1], [3], [5], [6], [7], [8]. Typically, a deep CNN is trained to construct a dense depth map given either a sparse depth input only or sparse depth aside with an RGB image.…”

Section: Introductionmentioning

confidence: 99%

“…The role of the network in both cases is to learn the manifold where the data live in. Since the publicly available datasets for scene depth completion such as the KITTI-Depth dataset [1] have a very high spatial resolution, the state-of-the-art methods [6], [7], [8] demand huge CNNs with millions of parameters to solve the problem. Unfortunately, this hinders the deployment of such methods in autonomous driving and robotic systems with limited computational and memory resources.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Confidence Propagation through CNNs for Guided Sparse Depth Regression

Eldesokey,

Felsberg,

Khan

2018

Preprint

View full text Add to dashboard Cite

Generally, convolutional neural networks (CNNs) process data on a regular grid, e.g. data generated by ordinary cameras. Designing CNNs for sparse and irregularly spaced input data is still an open research problem with numerous applications in autonomous driving, robotics, and surveillance. In this paper, we propose an algebraically-constrained normalized convolution layer for CNNs with highly sparse input that has a smaller number of network parameters compared to related work. We propose novel strategies for determining the confidence from the convolution operation and propagating it to consecutive layers. We also propose an objective function that simultaneously minimizes the data error while maximizing the output confidence. To integrate structural information, we also investigate fusion strategies to combine depth and RGB information in our normalized convolution network framework. In addition, we introduce the use of output confidence as an auxiliary information to improve the results. The capabilities of our normalized convolution network framework are demonstrated for the problem of scene depth completion. Comprehensive experiments are performed on the KITTI-Depth and the NYU-Depth-v2 datasets. The results clearly demonstrate that the proposed approach achieves superior performance while requiring only about 1-5% of the number of parameters compared to the state-of-the-art methods.

show abstract

Section: Related Workmentioning

confidence: 96%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Confidence Propagation through CNNs for Guided Sparse Depth Regression

Eldesokey,

Felsberg,

Khan

2018

Preprint

View full text Add to dashboard Cite

show abstract

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Zhu

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

218

142

View full text Add to dashboard Cite

Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse Li-DAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model. AbstractSorry about this little trick and hope it would work, since I do not have much time to neatten my original source code. depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a

show abstract

Distant Vehicle Detection Using Radar and Vision

Chadwick

Maddetn

Newman

2019

2019 International Conference on Robotics and Automation (ICRA)

162

View full text Add to dashboard Cite

For autonomous vehicles to be able to operate successfully they need to be aware of other vehicles with sufficient time to make safe, stable plans. Given the possible closing speeds between two vehicles, this necessitates the ability to accurately detect distant vehicles. Many current image-based object detectors using convolutional neural networks exhibit excellent performance on existing datasets such as KITTI. However, the performance of these networks falls when detecting small (distant) objects. We demonstrate that incorporating radar data can boost performance in these difficult situations. We also introduce an efficient automated method for training data generation using cameras of different focal lengths.

show abstract

Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation

Cited by 7 publications

References 0 publications

Confidence Propagation through CNNs for Guided Sparse Depth Regression

Confidence Propagation through CNNs for Guided Sparse Depth Regression

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

Distant Vehicle Detection Using Radar and Vision

Contact Info

Product

Resources

About