Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation

Abstract

Recent studies have witnessed that self-supervised methods based on view synthesis obtain clear progress on multi-view stereo (MVS). However, existing methods rely on the assumption that the corresponding points among different views share the same color, which may not always be true in practice. This may lead to unreliable self-supervised signal and harm the final reconstruction performance. To address the issue, we propose a framework integrated with more reliable supervision guided by semantic co-segmentation and data-augmentation. Specially, we excavate mutual semantic from multi-view images to guide the semantic consistency. And we devise effective data-augmentation mechanism which ensures the transformation robustness by treating the prediction of regular samples as pseudo ground truth to regularize the prediction of augmented samples. Experimental results on DTU dataset show that our proposed methods achieve the state-of-the-art performance among unsupervised methods, and even compete on par with supervised methods. Furthermore, extensive experiments on Tanks&Temples dataset demonstrate the effective generalization ability of the proposed method.

DS-MVSNet: Unsupervised Multi-view Stereo via Depth Synthesis

“…1, DS-MVSNet achieves the best accuracy, completeness, and overall score among coarse-to-fine supervised methods. Especially for completeness, the metric is improved by 10% compared with U-MVSNet-MS [30]. Fig.…”

“…However, these methods cannot be trained in an end-to-end manner. JDACS [30] proposed an end-to-end network, supervised by photometric consistency, segmentation map and augmentation data. However, it requires a pretrained feature extraction backbone for segmentation and inferring two times due to data augmentation.…”

“…Although these methods achieve superior performance, they cannot be trained in an end-to-end manner. Furthermore, some methods utilize additional inputs besides the multi-images, such as pre-processed optical flow [31], augmented data which infer two times [30] in training, and the pre-trained image semantic segmentation backbone [30]. These methods make the training more complicated and need more additional information.…”

Unsupervised 3D Reconstruction with Multi-Measure and High-Resolution Loss

“…The accuracy and completeness of the network reconstruction point cloud and the generalization ability of the network are better than most 3D reconstruction methods of the same period. Therefore, this method is widely used for depth estimation in most deep learning MVS networks [ 15 , 16 , 17 , 18 , 19 , 20 ]. However, to ensure the accuracy of depth calculation, the storage requirement is three times that of image resolution.…”

“…In this architecture, multi-scale pyramid feature aggregation is used to construct a 3D cost volume with more context information, and the loss function combines pixel loss and feature loss. In 2021, Xu et al [ 20 ] combined data augmentation and semantic segmentation as self-supervised signals, making the reconstruction effect comparable to that of the most advanced supervised learning networks. Yang et al [ 27 ] comprehensively used various methods such as deep fusion, mesh generation and deep rendering in unsupervised networks to optimize the pseudo depth.…”