MVS2: Deep Unsupervised Multi-View Stereo with Multi-View Symmetry

Abstract

The success of existing deep-learning based multi-view stereo (MVS) approaches greatly depends on the availability of large-scale supervision in the form of dense depth maps. Such supervision, while not always possible, tends to hinder the generalization ability of the learned models in never-seen-before scenarios. In this paper, we propose the first unsupervised learning based MVS network, which learns the multi-view depth maps from the input multi-view images and does not need ground-truth 3D training data. Our network is symmetric in predicting depth maps for all views simultaneously, where we enforce cross-view consistency of multi-view depth maps during both training and testing stages. Thus, the learned multi-view depth maps naturally comply with the underlying 3D scene geometry. Besides, our network also learns the multi-view occlusion maps, which further improves the robustness of our network in handling real-world occlusions. Experimental results on multiple benchmarking datasets demonstrate the effectiveness of our network and the excellent generalization ability.

Real-Time Unsupervised Multi-View Depth Estimation Network For Virtual View Synthesis

“…Moreover, due to the unideal run-time and memory requirements, the cascade pyramid structure [2] is proposed to build cost volume and infer depth in coarse to fine, which greatly reduces run-time and memory consumption. Besides, some unsupervised methods [7,3] are proposed to overcome the difficulty of obtaining ground-truth depth maps. These methods utilize photometric consistency and multi-view consistency to guide the training of the network.…”

“…Our method consists of three modules: multi-view preprocessing, an adaptive encoder-decoder network, postprocessing with shared occlusion masks, as shown in Figure 2. Inspired by present learning-based multi-view methods, such as [1,3], which use front-to-parallel planes at different depth as hypothesis planes, the first step of our pre-processing is to adopt homography to warp source images and then construct cost volumes, as shown in Figure 2. Given input sequences in the form of image-pose pairs, each of which contains a reference image I r for depth estimation, several additional source images {I i } N i=1 , and camera parameters.…”

“…Therefore, similar to existing unsupervised MVS methods [3,4], we reconstruct the target view to obtain the supervisory signal, which can be easily extended to different application scenarios like real-time depth estimation for virtual view synthesis. Moreover, previous work tends to utilize Fig.…”

“…So the proposed novel mask not only can deal with the occlusion problem but also can be used as a confidence fusion weight map to guide reliable matching. Compared with the occlusion processing strategies adopted by other unsupervised methods [7,3,4], our post-processing can effectively improve the accuracy and contrast of the esti-978-1-6654-4989-2/21/$31.00 ©2021 IEEE In this paper, we pursue real-time performance while improving the quality of depth estimation. We carefully adjusted an efficient and adaptive encoder-decoder network.…”

Cost Volume Pyramid Based Depth Inference for Multi-View Stereo

“…While it is possible to train the network using this synthetic data, for successfully deploying the model in real scenes, we still require to fine-tune the model using data from the target domain [16]. Another alternative is adopting an unsupervised learning strategy [6,13]. In this case, the few existing unsupervised MVS approaches use an image reconstruction loss to supervise the training process.…”