Visibility-Aware Point-Based Multi-View Stereo Network

Abstract

We introduce VA-Point-MVSNet, a novel visibility-aware point-based deep framework for multi-view stereo (MVS). Distinct from existing cost volume approaches, our method directly processes the target scene as point clouds. More specifically, our method predicts the depth in a coarse-to-fine manner. We first generate a coarse depth map, convert it into a point cloud and refine the point cloud iteratively by estimating the residual between the depth of the current iteration and that of the ground truth. Our network leverages 3D geometry priors and 2D texture information jointly and effectively by fusing them into a feature-augmented point cloud, and processes the point cloud to estimate the 3D flow for each point. This point-based architecture allows higher accuracy, more computational efficiency and more flexibility than cost-volume-based counterparts. Furthermore, our visibility-aware multi-view feature aggregation allows the network to aggregate multi-view appearance cues while taking into account occlusions. Experimental results show that our approach achieves a significant improvement in reconstruction quality compared with state-of-the-art methods on the DTU and the Tanks and Temples dataset. The code of VA-Point-MVSNet proposed in this work will be released at https://github.com/callmeray/PointMVSNet.

Cost Volume Pyramid Based Depth Inference for Multi-View Stereo

“…Another line of research [30,29,3,24] explore multiview aggregation to further leverage information from multiple view images and improve the performance of the network. Unlike these works mainly focusing on the backbone design or improving the view-aggregation strategy, we focus on self-supervised learning for depth inference.…”

“…We also compare our self-supervised results obtained without any ground-truth training data to traditional geometric-based MVS frameworks and supervised MVS networks, including recent methods with learning based view aggregation [29,3,30,24] that outperform our backbone network [25] in supervised scenario. Tab.…”

Fast and Robust 3D Reconstruction Solution from Permissive Open-Source Code

“…The PointMVSNet was able to produce a highquality and relatively fast recon struction, even without decreasing image resolution. After, Chen et al (2020) proposed the VisibilityAware (VAPoint MVSNet) extends the PointMVSNet with visibilityaware multiview feature aggregations, which aggregates informa tion to a better result with occlusions.…”

VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion

“…Occlusion-aware MVS: Occlusion detection with explicit photometric and geometric constraints has traditionally played an important role in MVS [19,32,34,35,36,47,54,56]. In addition, a number of MVS methods based on deep learning have proposed to learn visibility estimation [3,17,18,24]. Direct scene optimization: Yariv et al [50] propose to directly optimize the scene representation with respect to the input images.…”