HMS-Net: Hierarchical Multi-Scale Sparsity-Invariant Network for Sparse Depth Completion

Abstract

Dense depth cues are important and have wide applications in various computer vision tasks. In autonomous driving, LIDAR sensors are adopted to acquire depth measurements around the vehicle to perceive the surrounding environments. However, depth maps obtained by LIDAR are generally sparse because of its hardware limitation. The task of depth completion attracts increasing attention, which aims at generating a dense depth map from an input sparse depth map. To effectively utilize multi-scale features, we propose three novel sparsity-invariant operations, based on which, a sparsity-invariant multi-scale encoder-decoder network (HMS-Net) for handling sparse inputs and sparse feature maps is also proposed. Additional RGB features could be incorporated to further improve the depth completion performance. Our extensive experiments and component analysis on two public benchmarks, KITTI depth completion benchmark and NYU-depth-v2 dataset, demonstrate the effectiveness of the proposed approach. As of Aug. 12th, 2018, on KITTI depth completion leaderboard, our proposed model without RGB guidance ranks 1st among all peer-reviewed methods without using RGB information, and our model with RGB guidance ranks 2nd among all RGB-guided methods.

Depth Completion From Sparse LiDAR Data With Depth-Normal Constraints

“…Although we mainly focus on the outdoor application scenarios, we also train our model on indoor scenes, i.e., NYU-Depth-v2 dataset. As NYU-Depth-v2 dataset provides relatively denser depth measurements by Microsoft Kinect, we uniformly sample the depth map to obtain the sparser version following previous works [26,14]. We compare our results with latest CNN-based methods [26,2] as well as the classic methods [21,35,20] as shown in Table 3, and our method achieves state-of-the-art performance as well.…”

“…Ma et al concatenated the sparse depth and color image as the inputs of an off-the-shelf network [26] and further explored the feasibility of self-supervised Li-DAR completion [23]. Moreover, [14,16,33,4] proposed different network architectures to better exploit the potential of the encoder-decoder framework. However, the encoderdecoder architecture tends to predict the depth maps comprehensively but fails to concentrate on the local areas.…”

Unsupervised Depth Completion From Visual Inertial Odometry

“…Supervised Depth Completion minimizes the discrepancy between ground truth depth and depth predicted from an RGB image and sparse depth measurements. Methods focus on network topology [14,25,28], optimization [3,4,30], and modeling [5,8]. To handle sparse depth, [14] employed early fusion, where the image and sparse depth are convolved separately and the results concatenated as the input to a ResNet encoder.…”

“…[5] proposed a normalized convolutional layer to propagate sparse depth and used a binary validity map as a confidence measure. [8] proposed an upsampling layer and joint concatenation and convolution to deal with sparse inputs. All these methods require per-pixel ground-truth annotation.…”

“…As each sparse depth measurement can be viewed as a Dirac delta, [5,8,25] focused on propagating sparse depth through the network -a conventional convolution over the sparse depth input will give mostly zero activations. We, instead, circumvent this problem using our scaffolding.…”