Adaptive Context-Aware Multi-Modal Network for Depth Completion

Zhao, Shanshan; Gong, Maoguo; Fu, Huan; Tao, Dacheng

doi:10.1109/tip.2021.3079821

Cited by 125 publications

(84 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For dense estimation, Xiong et al [12] explored graph construction on 3D points to guide depth completion. Further, Zhao et al [36] directly constructed an encoder-decoder architecture based on graph propagation to explore better multi-modality feature fusion for depth completion. Although graph-based methods generally have fewer parameters, large-scale graph construction and messages aggregation require heavy computations, which limits general applications.…”

Section: Related Workmentioning

confidence: 99%

Depth Completion using Geometry-Aware Embedding

Wen¹,

Chen²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Exploiting internal spatial geometric constraints of sparse LiDARs is beneficial to depth completion, however, has been not explored well. This paper proposes an efficient method to learn geometry-aware embedding, which encodes the local and global geometric structure information from 3D points, e.g., scene layout, object's sizes and shapes, to guide dense depth estimation. Specifically, we utilize the dynamic graph representation to model generalized geometric relationship from irregular point clouds in a flexible and efficient manner. Further, we joint this embedding and corresponded RGB appearance information to infer missing depths of the scene with well structure-preserved details. The key to our method is to integrate implicit 3D geometric representation into a 2D learning architecture, which leads to a better trade-off between the performance and efficiency. Extensive experiments demonstrate that the proposed method outperforms previous works and could reconstruct fine depths with crisp boundaries in regions that are over-smoothed by them. The ablation study gives more insights into our method that could achieve significant gains with a simple design, while having better generalization capability and stability. The code is available at https://github.com/Wenchao-Du/GAENet.

show abstract

Section: Related Workmentioning

confidence: 99%

Depth Completion using Geometry-Aware Embedding

Wen¹,

Chen²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Besides, PI-RCNN fuses image features into point clouds with their new PACF module. Other methods, such as EPNet [27] and ACMNet [28], adopt a gating mechanism to control the portion of LiDAR and image information during fusion.…”

Section: A Multimodal Approaches For 3d Tasksmentioning

confidence: 99%

“…Although these previous works have demonstrated satisfactory results at the time they were published [28], [29], some later single-modal methods have surpassed them [30], [31]. Moreover, such extra performance often comes at the expense of higher computation and memory costs due to the extra inputs, thus incurring a higher hurdle for online tasks than their offline counterparts.…”

Section: A Multimodal Approaches For 3d Tasksmentioning

confidence: 99%

MAP-Gen: An Automated 3D-Box Annotation Flow with Multimodal Attention Point Generator

Liu¹,

Qian²,

Qi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Manually annotating 3D point clouds is laborious and costly, limiting the training data preparation for deep learning in real-world object detection. While a few previous studies tried to automatically generate 3D bounding boxes from weak labels such as 2D boxes, the quality is sub-optimal compared to human annotators. This work proposes a novel autolabeler, called multimodal attention point generator (MAP-Gen), that generates high-quality 3D labels from weak 2D boxes. It leverages dense image information to tackle the sparsity issue of 3D point clouds, thus improving label quality. For each 2D pixel, MAP-Gen predicts its corresponding 3D coordinates by referencing context points based on their 2D semantic or geometric relationships. The generated 3D points densify the original sparse point clouds, followed by an encoder to regress 3D bounding boxes. Using MAP-Gen, object detection networks that are weakly supervised by 2D boxes can achieve 94∼99% performance of those fully supervised by 3D annotations. It is hopeful this newly proposed MAP-Gen autolabeling flow can shed new light on utilizing multimodal information for enriching sparse point clouds.

show abstract

“…Except the application of spatial propagation scheme, Tang et al [19] proposed GuideNet to fuse the LiDAR data and RGB image information by performing learnable content-dependent and spatially-variant kernels. Zhao et al [25] proposed to adopt the graph propagation to model the observed spatial contexts with depth values, so as to better guide the recovery of the unobserved pixels' depth. More recently, PENet [11] and FCFR-Net [13] were proposed to carried out the depth completion through a two-stage coarse-to-fine mechanism and achieved impressive results.…”

Section: Depth Completion Based On Multimodal Learningmentioning

confidence: 99%

UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion

Chen¹,

Lin²,

Qin³

2022

Preprint

View full text Add to dashboard Cite

Depth prediction is a critical problem in robotics applications especially autonomous driving. Generally, depth prediction based on binocular stereo matching and fusion of monocular image and laser point cloud are two mainstream methods. However, the former usually suffers from overfitting while building cost volume, and the latter has a limited generalization due to the lack of geometric constraint.To solve these problems, we propose a novel multimodal neural network, namely UAMD-Net, for dense depth completion based on fusion of binocular stereo matching and the weak constrain from the sparse point clouds. Specifically, the sparse point clouds are converted to sparse depth map and sent to the multimodal feature encoder (MFE) with binocular image, constructing a cross-modal cost volume. Then, it will be further processed by the multimodal feature aggregator (MFA) and the depth regression layer. Furthermore, the existing multimodal methods ignore the problem of modal dependence, that is, the network will not work when a certain modal input has a problem. Therefore, we propose a new training strategy called Modal-dropout which enables the network to be adaptively trained with multiple modal inputs and inference with specific modal inputs. Benefiting from the flexible network structure and adaptive training method, our proposed network can realize unified training under various modal input conditions. Comprehensive experiments conducted on KITTI depth completion benchmark demonstrate that our method produces robust results and outperforms other stateof-the-art methods.

show abstract

Adaptive Context-Aware Multi-Modal Network for Depth Completion

Cited by 125 publications

References 64 publications

Depth Completion using Geometry-Aware Embedding

Depth Completion using Geometry-Aware Embedding

MAP-Gen: An Automated 3D-Box Annotation Flow with Multimodal Attention Point Generator

UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion

Contact Info

Product

Resources

About