Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image

Abstract

We consider the problem of dense depth prediction from a sparse set of depth measurements and a single RGB image. Since depth estimation from monocular images alone is inherently ambiguous and unreliable, to attain a higher level of robustness and accuracy, we introduce additional sparse depth samples, which are either acquired with a low-resolution depth sensor or computed via visual Simultaneous Localization and Mapping (SLAM) algorithms. We propose the use of a single deep regression network to learn directly from the RGB-D raw data, and explore the impact of number of depth samples on prediction accuracy. Our experiments show that, compared to using only RGB images, the addition of 100 spatially random depth samples reduces the prediction root-mean-square error by 50% on the NYU-Depth-v2 indoor dataset. It also boosts the percentage of reliable prediction from 59% to 92% on the KITTI dataset. We demonstrate two applications of the proposed algorithm: a plug-in module in SLAM to convert sparse maps to dense maps, and super-resolution for LiDARs. Software 2 and video demonstration 3 are publicly available.

Robust Full-Fov Depth Estimation in Tele-Wide Camera System

“…6, respectively. Even though the training and test data are not the same across various methods, the scenes are similar because they are captured during driving and from the same sensor [20]. We can see that our model with the support of tele-FoV depth can boost RMSE, REL and δ values a lot, and also achieve better subjective quality.…”

Predicting Unobserved Space for Planning via Depth Map Augmentation

“…We retrain a depth augmentation method based on the residual neural network proposed by [4] motivated by its performance on predicting depth from even sparser depth samples on the NYU-Depth-v2 dataset. [4] has a similar architecture as in previous work [9] which in turn is based on the ResNet-50 architecture introduced in [26].…”

“…For predicting complete depth maps we train a network based on the residual neural network proposed by [4]. As input layer we use a four-channel image of size 320x240 instead of the 308x224 used in [4]. As suggested in [4] we use the mean absolute error (L 1 ) as loss function.…”

“…As input layer we use a four-channel image of size 320x240 instead of the 308x224 used in [4]. As suggested in [4] we use the mean absolute error (L 1 ) as loss function. To improve robustness, we augment the input data by flipping the image horizontally with 0.5 chance and adjust brightness, contrast and saturation by factors randomly chosen between 0.6 and 1.4.…”

“…Therefore -inspired by the human vision system -we propose a system that tackles this problem by augmenting these sparse depth maps with predictions based on the previously learned shape of the world. We retrain the residual neural network proposed by [4] to augment sparse depth maps by predicting unknown pixels from color images. To that end, we emulate commonly occurring failures of real depth sensors, depth estimation, and mapping techniques such as missing depth measurements in low-texture areas, to generate training data.…”