Deep Architecture With Cross Guidance Between Single Image and Sparse LiDAR Data for Depth Completion

Abstract

It is challenging to apply depth maps generated from sparse laser scan data to computer vision tasks, such as robot vision and autonomous driving, because of the sparsity and noise in the data. To overcome this problem, depth completion tasks have been proposed to produce a dense depth map from sparse LiDAR data and a single RGB image. In this study, we developed a deep convolutional architecture with cross guidance for multi-modal feature fusion to compensate for the lack of representation power of their modality. Two encoders, which are part of the proposed architecture, receive different modalities as inputs. They interact with each other by exchanging information in each stage through the attention mechanism during encoding. We also propose a residual atrous spatial pyramid block, comprising multiple dilated convolutions with different dilation rates, which are used to derive highly significant features. The experimental results of the KITTI depth completion benchmark dataset demonstrate that the proposed architecture shows higher performance than that of the other models trained in a two-dimensional space without pre-training or finetuning other datasets. INDEX TERMS Depth estimation, depth completion, LiDAR data, cross guidance, multi-scale dilated convolutional block. Recently, artificial neural network models with deep learning have been used in state-of-the-art technologies of pattern recognition and machine learning. In particular, convolutional neural networks (CNNs) exhibit excellent performance in many computer vision tasks. While conventional CNNs [3]-[5] comprise blocks of stacking convolution

DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

“…CFCNet [8] learns to capture the semantically correlated features between RGB and depth information. CG-Net [9] proposes a cross guidance module to fuse the multi-modal feature from RGB and LiDAR. Some methods rely on iterative Spatial Propagation Network (SPN) to better treat the difficulties made by the sparsity and irregular distribution of the input [10][11][12] [13].…”

BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning

“…Some methods, e.g. [45], fuse the sparse depth and RGB image via early fusion while others [18,26,29,37,44,64] utilize a late fusion scheme, or jointly utilize both the early and late fusion [36,65,68]. Another line of research focuses on utilizing affinity or geometric information of the scene via surface normal, occlusion boundaries, and the geometric convolutional layer [11,12,25,34,52,54,77,86].…”

DVMN: Dense Validity Mask Network for Depth Completion

“…These solutions train their normal prediction with synthetic data [10] or based on principal component analysis [11]. Others use confidence to fuse image and depth features, giving more weight to the modality with less uncertainty [12], [13], [14].…”

“…Various networks integrate Spatial Pyramid Pooling (SPP) [22] for depth completion [5], [16], [18], [23]. Atrous Spatial Pyramid Pooling (ASPP) [24] has been studied at the end of an encoder [16] or within residual blocks [13]. Li et al [25] combine multiple networks, each using different resolutions of sparse input.…”