Sparsity Invariant CNNs

Uhrig, Jonas; Schneider, N. M.; Schneider, Lukas; Franke, Uwe; Brox, Thomas; Geiger, Andreas

doi:10.48550/arxiv.1708.06500

Cited by 11 publications

(45 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we address all these challenges with two contributions: (1) We develop a network architecture that is able to learn a direct mapping from the sparse depth (and color images, if available) to dense depth. This architecture achieves state-of-the-art accuracy on the KITTI Depth Completion Benchmark [4] and is currently the leading method. propose a self-supervised framework for training depth completion networks.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera

Cavalheiro

Karaman

2019

2019 International Conference on Robotics and Automation (ICRA)

385

576

View full text Add to dashboard Cite

Depth completion, the technique of estimating a dense depth image from sparse depth measurements, has a variety of applications in robotics and autonomous driving. However, depth completion faces 3 main challenges: the irregularly spaced pattern in the sparse depth input, the difficulty in handling multiple sensor modalities (when color images are available), as well as the lack of dense, pixel-level ground truth depth labels. In this work, we address all these challenges. Specifically, we develop a deep regression model to learn a direct mapping from sparse depth (and color images) to dense depth. We also propose a self-supervised training framework that requires only sequences of color and sparse depth images, without the need for dense depth labels. Our experiments demonstrate that our network, when trained with semi-dense annotations, attains state-of-theart accuracy and is the winning approach on the KITTI depth completion benchmark 2 at the time of submission. Furthermore, the self-supervised framework outperforms a number of existing solutions trained with semidense annotations.

show abstract

Section: Introductionmentioning

confidence: 99%

Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera

Cavalheiro

Karaman

2019

2019 International Conference on Robotics and Automation (ICRA)

385

576

View full text Add to dashboard Cite

show abstract

“…Convolution For imaging data, hierarchical convolutional operations extract meaningful representations. Existing methods often fill in missing values with a constant, such as zero or the mean value at that pixel across the dataset, but these do not usually accurately estimate the existing image signal [47]. Given a sparse input image, we experiment with two convolutional approaches.…”

Section: Learning Via Sparsity Aware Neural Networkmentioning

confidence: 99%

“…Given a sparse input image, we experiment with two convolutional approaches. In the first, we apply a weighted convolution, where the convolution kernel c is modified to vary with image location k by the binary observed mask m [47]. The new filter response r at location k is…”

Section: Learning Via Sparsity Aware Neural Networkmentioning

confidence: 99%

“…Since convolutional layers can be applied hierarchically, even very sparse data will often lead to a dense deep feature response. This sort of convolution was recently used to impute LIDAR depth data in a supervised context [47].…”

Section: Learning Via Sparsity Aware Neural Networkmentioning

confidence: 99%

“…Sensor failure, occlusions, or sparsity by design all lead to missing data. For example, LIDAR scan data, providing depth measurements in a variety of problems, yield sparse point clouds [5,28,47]. Our work is motivated by a challenging real world medical imaging problem.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Data Imputation via Variational Inference of Deep Subspaces

Dalca¹,

Guttag²,

Sabuncu³

2019

Preprint

View full text Add to dashboard Cite

A wide range of systems exhibit high dimensional incomplete data. Accurate estimation of the missing data is often desired, and is crucial for many downstream analyses. Many state-of-the-art recovery methods involve supervised learning using datasets containing full observations. In contrast, we focus on unsupervised estimation of missing image data, where no full observations are availablea common situation in practice. Unsupervised imputation methods for images often employ a simple linear subspace to capture correlations between data dimensions, omitting more complex relationships. In this work, we introduce a general probabilistic model that describes sparse high dimensional imaging data as being generated by a deep nonlinear embedding. We derive a learning algorithm using a variational approximation based on convolutional neural networks and discuss its relationship to linear imputation models, the variational auto encoder, and deep image priors. We introduce sparsity-aware network building blocks that explicitly model observed and missing data. We analyze proposed sparsity-aware network building blocks, evaluate our method on public domain imaging datasets, and conclude by showing that our method enables imputation in an important real-world problem involving medical images. The code is freely available as part of the neuron library at http://github.com/adalca/neuron.

show abstract

Image Inpainting for Irregular Holes Using Partial Convolutions

Liu

Reda

Shih

et al. 2018

Computer Vision – ECCV 2018

1,545

1,796

View full text Add to dashboard Cite

Fig. 1. Masked images and corresponding inpainted results using our partialconvolution based network.Abstract. Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Postprocessing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.

show abstract

Sparsity Invariant CNNs

Cited by 11 publications

References 0 publications

Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera

Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera

Unsupervised Data Imputation via Variational Inference of Deep Subspaces

Image Inpainting for Irregular Holes Using Partial Convolutions

Contact Info

Product

Resources

About