Multi-modal Masked Pre-training for Monocular Panoramic Depth Completion

Yan, Zhiqiang; Li, Xiang; Wang, Kun; Zhang, Zhenyu; Li, Jun; Yang, Jian

doi:10.1007/978-3-031-19769-7_22

Cited by 8 publications

(5 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4 reports the comparison on two panoramic depth completion datasets. We observe that, with the lowest parameters, RigNet++ is still significantly superior to UniFuse [30], HoHoNet [59], GuideNet [63], 360Depth [53], and M 3 PT [71]. For example, compared to M 3 PT that uses additional masked pre-training strategy [19], RigNet++ still achieves 15.1% lower RMSE in average and higher δ i despite their REL metrics are marginally close.…”

Section: Evaluation On Indoor Nyuv2mentioning

confidence: 87%

“…The latest Mat-terport3D 1 (512 × 256) contains 7,907 panoramic RGB-D pairs, of which 5636 for training, 744 for validating, and 1527 for testing. For panoramic depth completion, M 3 PT [71] proposes to synthesize the sparse depth. It first projects the equirectangular ground-truth depth into cubical map to remove the distortion.…”

Section: Datasets and Metricsmentioning

confidence: 99%

“…Depth completion, the technique of converting sparse depth measurements to dense ones, has a variety of applications in the computer vision field, such as autonomous driving [18,8,66,74], augmented reality [9,58,72], virtual reality [2], and 3D scene reconstruction [48,56,71]. The success of these applications heavily depends on reliable depth predictions.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

RigNet: Repetitive Image Guided Network for Depth Completion

Yan

Wang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Depth completion aims to recover dense depth maps from sparse ones, where color images are often used to facilitate this task. Recent depth methods primarily focus on image guided learning frameworks. However, blurry guidance in the image and unclear structure in the depth still impede their performance. To tackle these challenges, we explore an efficient repetitive design in our image guided network to gradually and sufficiently recover depth values. Specifically, the efficient repetition is embodied in both the image guidance branch and depth generation branch. In the former branch, we design a dense repetitive hourglass network to extract discriminative image features of complex environments, which can provide powerful contextual instruction for depth prediction. In the latter branch, we introduce a repetitive guidance module based on dynamic convolution, in which an efficient convolution factorization is proposed to reduce the complexity while modeling high-frequency structures progressively. Extensive experiments indicate that our approach achieves superior or competitive results on KITTI, VKITTI, NYUv2, 3D60, and Matterport3D datasets.

show abstract

Section: Evaluation On Indoor Nyuv2mentioning

confidence: 87%

Section: Datasets and Metricsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

RigNet: Repetitive Image Guided Network for Depth Completion

Yan

Wang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Most of the existing middle fusion methods only exploited simple concatenation or summation operation, which cannot effectively fuse multi-modal information. Some recent works had explored several effective fusion methods, such as guided convolution module [16], multi-modal masked pre-training (M3PT) [24], adaptive symmetric gated fusion [17] and channel shuffle [20]. Although PENet [22] considered both the early and late fusion, they only used simple concatenate operations.…”

Section: Multi-modal Fusionmentioning

confidence: 99%

“…Considering the fact that RGB data modality and depth data modality have different statistical properties, various fusion methods [4,[15][16][17][18][19][20][21][22][23][24][25] have been proposed to eliminate the modal distinction to better fuse RGB and depth information. Jaritz et al [15] demonstrated that the middle fusion performed better than the early fusion strategy.…”

Section: Introductionmentioning

confidence: 99%

SPNet: Structure preserving network for depth completion

Luo

Fan

et al. 2023

PLoS ONE

View full text Add to dashboard Cite

Depth completion aims to predict a dense depth map from a sparse one. Benefiting from the powerful ability of convolutional neural networks, recent depth completion methods have achieved remarkable performance. However, it is still a challenging problem to well preserve accurate depth structures, such as tiny structures and object boundaries. To tackle this problem, we propose a structure preserving network (SPNet) in this paper. Firstly, an efficient multi-scale gradient extractor (MSGE) is proposed to extract useful multi-scale gradient images, which contain rich structural information that is helpful in recovering accurate depth. The MSGE is constructed based on the proposed semi-fixed depthwise separable convolution. Meanwhile, we adopt a stable gradient MAE loss (LGMAE) to provide additional depth gradient constrain for better structure reconstruction. Moreover, a multi-level feature fusion module (MFFM) is proposed to adaptively fuse the spatial details from low-level encoder and the semantic information from high-level decoder, which will incorporate more structural details into the depth modality. As demonstrated by experiments on NYUv2 and KITTI datasets, our method outperforms some state-of-the-art methods in terms of both quantitative and quantitative evaluations.

show abstract

PanoFormer: Panorama Transformer for Indoor 360$$^{\circ }$$ Depth Estimation

Shen

Lin

Liao

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Existing panoramic layout estimation solutions tend to recover room boundaries from a vertically compressed sequence, yielding imprecise results as the compression process often muddles the semantics between various planes. Besides, these datadriven approaches impose an urgent demand for massive data annotations, which are laborious and time-consuming. For the first problem, we propose an orthogonal plane disentanglement network (termed DOPNet) to distinguish ambiguous semantics. DOPNet consists of three modules that are integrated to deliver distortion-free, semantics-clean, and detail-sharp disentangled representations, which benefit the subsequent layout recovery. For the second problem, we present an unsupervised adaptation technique tailored for horizon-depth and ratio representations. Concretely, we introduce an optimization strategy for decision-level layout analysis and a 1D cost volume construction method for feature-level multi-view aggregation, both of which are designed to fully exploit the geometric consistency across multiple perspectives. The optimizer provides a reliable set of pseudo-labels for network training, while the 1D cost volume enriches each view with comprehensive scene information derived from other perspectives. Extensive experiments demonstrate that our solution outperforms other SoTA models on both monocular layout estimation and multi-view layout estimation tasks.

show abstract

Multi-modal Masked Pre-training for Monocular Panoramic Depth Completion

Cited by 8 publications

References 59 publications

RigNet: Repetitive Image Guided Network for Depth Completion

RigNet: Repetitive Image Guided Network for Depth Completion

SPNet: Structure preserving network for depth completion

PanoFormer: Panorama Transformer for Indoor 360$$^{\circ }$$ Depth Estimation

Contact Info

Product

Resources

About