To Complete or to Estimate, That is the Question: A Multi-Task Approach to Depth Completion and Monocular Depth Estimation

Abstract

Robust three-dimensional scene understanding is now an ever-growing area of research highly relevant in many realworld applications such as autonomous driving and robotic navigation. In this paper, we propose a multi-task learningbased model capable of performing two tasks:-sparse depth completion (i.e. generating complete dense scene depth given a sparse depth image as the input) and monocular depth estimation (i.e. predicting scene depth from a single RGB image) via two sub-networks jointly trained end to end using data randomly sampled from a publicly available corpus of synthetic and real-world images. The first subnetwork generates a sparse depth image by learning lower level features from the scene and the second predicts a full dense depth image of the entire scene, leading to a better geometric and contextual understanding of the scene and, as a result, superior performance of the approach. The entire model can be used to infer complete scene depth from a single RGB image or the second network can be used alone to perform depth completion given a sparse depth input. Using adversarial training, a robust objective function, a deep architecture relying on skip connections and a blend of synthetic and real-world training data, our approach is capable of producing superior high quality scene depth. Extensive experimental evaluation demonstrates the efficacy of our approach compared to contemporary state-of-the-art techniques across both problem domains.

Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data

“…Following synthetic-to-real depth estimation methods [41,42], we use as a domain adaptation baseline a CycleGAN [1] to adapt the images. To sparsify the synthetic depth, we use the real LiDAR masks [43], shown in Table 1 to perform better than Bernoulli sampling. The performance of this domain adaptation baseline is presented in Table 3 in DA Base.…”

“…For monocular depth estimation, two domain adaptation approaches used style-transfer methods [41,42]. Sparse-to-dense methods, however, have used synthetic data without any adaptation [4,6,43] so far. Training on synthetic data requires a high rendering quality [44].…”

“…We now propose an approach to address these two issues. from the real domain similarly to [43]. We use x s R,D to generate a binary mask…”

Improving Depth Estimation by Embedding Semantic Segmentation: A Hybrid CNN Model

“…Finally, Koch et al [12] make an analysis and comparisons between all the methods mentioned before. Atapour-Abarghouei and Breckon [13] use an arrangement of eight CNN models (U-Net) [14] that first estimates the semantic segmentation of the image and then estimates depth from the segmented objects. Lin et al [15] proposed an architecture that joins a CNN that estimates depth and separately, a CNN that estimates the semantic segmentation.…”

Adaptive Context-Aware Multi-Modal Network for Depth Completion

“…Currently, CNNs have been a dominant solution for depth completion [21,27,28,29,30,31,32,19,33,34,17,18,35,36], outperforming traditional methods by a wide margin. In specific, to learn representations of the irregular and sparse LiDAR data, Uhrig et al [9] proposed the sparsity-invariant convolutional operation.…”