DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion

Abstract

In this paper we propose a convolutional neural network that is designed to upsample a series of sparse range measurements based on the contextual cues gleaned from a high resolution intensity image. Our approach draws inspiration from related work on super-resolution and in-painting. We propose a novel architecture that seeks to pull contextual cues separately from the intensity image and the depth features and then fuse them later in the network. We argue that this approach effectively exploits the relationship between the two modalities and produces accurate results while respecting salient image structures. We present experimental results to demonstrate that our approach is comparable with state of the art methods and generalizes well across multiple datasets.1

To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

“…Error Our Approach 892 243 Table 2: Comparison of our depth completion approach against [10,15,16,39,50,54,55] using the validation set in [54]. Despite not being the primary focus, our completion approach remains competitive with the state of the art.…”

“…Even though sparse depth completion is not the primary objective of this work, our approach is capable of generating dense depth from a sparse input along with its primary function (monocular depth estimation) and can outperform a variety of prior related work [10,16,40,50,54].…”

“…Our sparse generator follows an encoder/decoder architecture with every layer containing modules of convolution, BatchNorm and leaky ReLU (slope = 0.2) with skip connections [46] between every pair of corresponding layers in the encoder and the decoder (Figure 2 -SG). The Figure 5: Comparing our depth completion results against [15,40,54,50]. The depth images have been adjusted for better visualization.…”

“…The depth images have been adjusted for better visualization. RGB: input colour image; LiDAR: sparse depth input; SIC: sparsity invariant CNN [54]; StD: sparse to dense completion [40]; DFN: deep fusion network [50]; GSR: guided sparse regression [15]; Dense: dense ground truth from [54]. dense generator follows a somewhat similar architecture, save that in its encoder, residual blocks [22] form each layer, with the output from each passed to corresponding decoding layers via skip connections.…”

“…• Monocular depth estimation via adversarial training, a deep architecture with skip connections and a robust compound objective function directly supervised using this framework to outperform prior contemporary work [5,7,14,20,31,36,62,66]. • Sparse to dense depth completion via the same multitask model, capable of generating a dense depth output given a sparse depth input captured via a LiDAR sensor with results superior to prior contemporary work [10,16,40,50,54]. • Unique leverage of both synthetic [17] and real-world datasets [54] to ensure high-density complete depth outputs, despite such levels of density not existing in any real-world training images.…”