Adaptive Context-Aware Multi-Modal Network for Depth Completion

Zhao, Shanshan; Gong, Mingming; Fu, Huan; Tao, Dacheng

doi:10.48550/arxiv.2008.10833

Cited by 7 publications

(9 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GNNs have been applied in various vision tasks, such as image classification [12,47], object detection [19,17] and visual question answering [44,34]. Unlike previous depth completion methods using GNNs for multi-modality fusion [51], learning dynamic kernel [49], we leverage GNNs as its message passing principle is in accord with spatial propagation. In addition, we develop a geometry-aware and dynamically constructed GCN with edge attention to aggregate and update information from neighboring nodes.…”

Section: Spatial Propagation Networkmentioning

confidence: 99%

GraphCSPN: Geometry-Aware Depth Completion via Dynamic GCNs

Liu

Shao²,

Wang³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Image guided depth completion aims to recover per-pixel dense depth maps from sparse depth measurements with the help of aligned color images, which has a wide range of applications from robotics to autonomous driving. However, the 3D nature of sparse-to-dense depth completion has not been fully explored by previous methods. In this work, we propose a Graph Convolution based Spatial Propagation Network (GraphCSPN) as a general approach for depth completion. First, unlike previous methods, we leverage convolution neural networks as well as graph neural networks in a complementary way for geometric representation learning. In addition, the proposed networks explicitly incorporate learnable geometric constraints to regularize the propagation process performed in three-dimensional space rather than in two-dimensional plane. Furthermore, we construct the graph utilizing sequences of feature patches, and update it dynamically with an edge attention module during propagation, so as to better capture both the local neighboring features and global relationships over long distance. Extensive experiments on both indoor NYU-Depth-v2 and outdoor KITTI datasets demonstrate that our method achieves the state-of-the-art performance, especially when compared in the case of using only a few propagation steps. Code and models are available at the project page 3 .

show abstract

Section: Spatial Propagation Networkmentioning

confidence: 99%

GraphCSPN: Geometry-Aware Depth Completion via Dynamic GCNs

Liu

Shao²,

Wang³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…So far various strategies have been developed to encode geometric cues. For instance, Uber-ATG [20] applies continuous convolution on 3D points, ACMNet [21] exploits the graph propagation, DeepLi-DAR [5] and PwP [17] use surface normal to introduce geometric constraints. These methods are either complicated in computation or need extra data for learning.…”

Section: B Geometric Encodingmentioning

confidence: 99%

PENet: Towards Precise and Efficient Image Guided Depth Completion

Hu¹,

Wang²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Image guided depth completion is the task of generating a dense depth map from a sparse depth map and a high quality image. In this task, how to fuse the color and depth modalities plays an important role in achieving good performance. This paper proposes a two-branch backbone that consists of a color-dominant branch and a depth-dominant branch to exploit and fuse two modalities thoroughly. More specifically, one branch inputs a color image and a sparse depth map to predict a dense depth map. The other branch takes as inputs the sparse depth map and the previously predicted depth map, and outputs a dense depth map as well. The depth maps predicted from two branches are complimentary to each other and therefore they are adaptively fused. In addition, we also propose a simple geometric convolutional layer to encode 3D geometric cues. The geometric encoded backbone conducts the fusion of different modalities at multiple stages, leading to good depth completion results. We further implement a dilated and accelerated CSPN++ to refine the fused depth map efficiently. The proposed full model ranks 1st in the KITTI depth completion online leaderboard at the time of submission. It also infers much faster than most of the top ranked methods. The code of this work is available at https://github.com/JUGGHM/PENet_ICRA2021.

show abstract

“…These solutions train their normal prediction with synthetic data [10] or based on principal component analysis [11]. Others use confidence to fuse image and depth features, giving more weight to the modality with less uncertainty [12], [13], [14].…”

Section: A State Of the Art -Depth Completionmentioning

confidence: 99%

DVMN: Dense Validity Mask Network for Depth Completion

Reichardt¹,

Mangat²,

Wasenmüller³

2021

Preprint

View full text Add to dashboard Cite

LiDAR depth maps provide environmental guidance in a variety of applications. However, such depth maps are typically sparse and insufficient for complex tasks such as autonomous navigation. State of the art methods use image guided neural networks for dense depth completion. We develop a guided convolutional neural network focusing on gathering dense and valid information from sparse depth maps. To this end, we introduce a novel layer with spatially variant and content-depended dilation to include additional data from sparse input. Furthermore, we propose a sparsity invariant residual bottleneck block. We evaluate our Dense Validity Mask Network (DVMN) on the KITTI depth completion benchmark and achieve state of the art results. At the time of submission, our network is the leading method using sparsity invariant convolution.

show abstract

Adaptive Context-Aware Multi-Modal Network for Depth Completion

Cited by 7 publications

References 67 publications

GraphCSPN: Geometry-Aware Depth Completion via Dynamic GCNs

GraphCSPN: Geometry-Aware Depth Completion via Dynamic GCNs

PENet: Towards Precise and Efficient Image Guided Depth Completion

DVMN: Dense Validity Mask Network for Depth Completion

Contact Info

Product

Resources

About