Dense Depth Posterior (DDP) From Single Image and Sparse Range

Abstract

We present a deep learning system to infer the posterior distribution of a dense depth map associated with an image, by exploiting sparse range measurements, for instance from a lidar. While the lidar may provide a depth value for a small percentage of the pixels, we exploit regularities reflected in the training set to complete the map so as to have a probability over depth for each pixel in the image. We exploit a Conditional Prior Network, that allows associating a probability to each depth value given an image, and combine it with a likelihood term that uses the sparse measurements. Optionally we can also exploit the availability of stereo during training, but in any case only require a single image and a sparse point cloud at run-time. We test our approach on both unsupervised and supervised depth completion using the KITTI benchmark, and improve the state-ofthe-art in both.

Unsupervised Depth Completion From Visual Inertial Odometry

“…Compared to [14] and [28], our VGG11 model has a 76.1% and 61.5% reduction in the encoder parameters and 65.1% and 48.4% overall, respectively. Our VGG8 model has a 89.9% and 83.9% reduction in the encoder and 80% and 66% overall compared to that of [14] and [28], respectively. Despite having fewer parameters, our method outperforms that of [14,28].…”

“…This is in contrast to other unsupervised methods [14] (who follows early fusion and concatenates features from the two branches after the first convolution) and [28] (late fusion) -both of whom use ResNet34 encoders with ≈ 23.8M and ≈ 14.8M parameters, respectively. Both [14,28] employ the same decoder with ≈ 4M parameterstotaling to ≈ 27.8M and ≈ 18.8M parameters, respectively. Compared to [14] and [28], our VGG11 model has a 76.1% and 61.5% reduction in the encoder parameters and 65.1% and 48.4% overall, respectively.…”

“…[22] used a local smoothness term, but instead minimized the photometric error between rectified stereo-pairs where pose is known. [28] also leveraged stereo pairs and a more sophisticated photometric loss (SSIM [27]), and replaced the generic smoothness term with a conditional prior to measure compatibility between the prediction and a learned depth model obtained by training a separate network on ground-truth depth. This method can be considered semi-unsupervised, and requires ground truth for training the prior.…”

“…We propose two encoder-decoder architectures with skip connections following the late fusion paradigm [9,28]. Each encoder has an image branch and a depth branch -the image branch contains 75% of the total features in the encoder and the depth branch 25%.…”

“…Both [14,28] employ the same decoder with ≈ 4M parameterstotaling to ≈ 27.8M and ≈ 18.8M parameters, respectively. Compared to [14] and [28], our VGG11 model has a 76.1% and 61.5% reduction in the encoder parameters and 65.1% and 48.4% overall, respectively. Our VGG8 model has a 89.9% and 83.9% reduction in the encoder and 80% and 66% overall compared to that of [14] and [28], respectively.…”