Vis-MVSNet: Visibility-Aware Multi-view Stereo Network

Zhang, Jingyang; Li, Shiwei; Luo, Zixin; Fang, Tian; Yao, Yao

doi:10.1007/s11263-022-01697-3

Cited by 83 publications

(35 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(mm) Comp. (mm) Overall(mm) COLMAP [15] 0.400 0.664 0.532 MVSNet [7] 0.396 0.527 0.462 R-MVSNet [10] 0.385 0.459 0.422 D 2 HC-RMVSNet [11] 0.395 0.378 0.386 PointMVSNet [16] 0.342 0.411 0.376 Vis-MVSNet [9] 0.369 0.361 0.365 AA-RMVSNet [1] 0.376 0.339 0.357 CasMVSNet [8] 0.325 0.385 0.355 EPP-MVSNet [17] 0.413 0.296 0.355 PatchmatchNet [18] 0.427 0.277 0.352 UCS-Net [19] 0.338 0.349 0.344 Ours+CasMVSNet 0.323(-0.003) 0.347(-0.038) 0.335(-0.020) the mean absolute distance of all pairs is the method of Accuracy measures, while the Completeness measures is the opposite. At last, the overall performance of models is the average of Accuracy and Completeness measures.…”

Section: Methodsmentioning

confidence: 99%

“…MVSNet [7] transforms the MVS task to a per-view depth map estimation task that encodes camera parameters via differentiable homography to build 3D cost volumes, which will be regularized by 3D CNN to obtain a probability volume and final depth. Inspired by MVSNet [7], [8,9] follow this design paradigm. However, 3D U-Net architecture costs a lot of memory and runtime for cost volume regularization.…”

Section: Learning Based Multi-view Stereomentioning

confidence: 99%

See 1 more Smart Citation

Enhancing Multi-View Stereo with Contrastive Matching and Weighted Focal Loss

Ding

Huang

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Learning-based multi-view stereo (MVS) methods have made impressive progress and surpassed traditional methods in recent years. However, their accuracy and completeness are still struggling. In this paper, we propose a new method to enhance the performance of existing networks inspired by contrastive learning and feature matching. First, we propose a Contrast Matching Loss (CML), which treats the correct matching points in depth-dimension as positive sample and other points as negative samples, and computes the contrastive loss based on the similarity of features. We further propose a Weighted Focal Loss (WFL) for better classification capability, which weakens the contribution of lowconfidence pixels in unimportant areas to the loss according to predicted confidence. Extensive experiments performed on DTU, Tanks and Temples and BlendedMVS datasets show our method achieves state-of-the-art performance and significant improvement over baseline network.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Learning Based Multi-view Stereomentioning

confidence: 99%

Enhancing Multi-View Stereo with Contrastive Matching and Weighted Focal Loss

Ding

Huang

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…Previous works adopted various solutions to learn mutual correlations [85], avoiding the influences of incorrect matches caused by occlusion. Popular solutions include the visibility-based aggregation [8], [80], the attention-based aggregation [66], [73], [77], etc. RayMVSNet follows the attention-based aggregation route.…”

Section: Related Workmentioning

confidence: 99%

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

Shi²,

Wang

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Learning-based multi-view stereo (MVS) has by far centered around 3D convolution on cost volumes. Due to the high computation and memory consumption of 3D CNN, the resolution of output depth is often considerably limited. Different from most existing works dedicated to adaptive refinement of cost volumes, we opt to directly optimize the depth value along each camera ray, mimicking the range (depth) finding of a laser scanner. This reduces the MVS problem to ray-based depth optimization which is much more light-weight than full cost volume optimization. In particular, we propose RayMVSNet which learns sequential prediction of a 1D implicit field along each camera ray with the zero-crossing point indicating scene depth. This sequential modeling, conducted based on transformer features, essentially learns the epipolar line search in traditional multi-view stereo. We devise a multi-task learning for better optimization convergence and depth accuracy. We found the monotonicity property of the SDFs along each ray greatly benefits the depth estimation. Our method ranks top on both the DTU and the Tanks & Temples datasets over all previous learning-based methods, achieving an overall reconstruction score of 0.33mm on DTU and an F-score of 59.48% on Tanks & Temples. It is able to produce high-quality depth estimation and point cloud reconstruction in challenging scenarios such as objects/scenes with non-textured surface, severe occlusion, and highly varying depth range. Further, we propose RayMVSNet++ to enhance contextual feature aggregation for each ray through designing an attentional gating unit to select semantically relevant neighboring rays within the local frustum around that ray. This improves the performance on datasets with more challenging examples (e.g. low-quality images caused by poor lighting conditions or motion blur). RayMVSNet++ achieves state-of-the-art performance on the ScanNet dataset. In particular, it attains an AbsRel of 0.058m and produces accurate results on the two subsets of textureless regions and large depth variation.

show abstract

“…The fact that even with every day photogrammetric sensors we can retrieve point cloud representations is of huge impact. The clouds can be calculated from multiple un-referenced images of various point of views inside a scenery (Kuhn et al, 2020;Zhang et al, 2020). Even monocular methods for producing highly accurate depth maps from single photographies exist (Alhashim and Wonka, 2018;Yuan et al, 2022).…”

Section: Technological Outlookmentioning

confidence: 99%

Improving quantitative structure models of trees inspired by pipe and metabolic scaling theory

Hackenberg¹,

Bontemps²

2022

Preprint

View full text Add to dashboard Cite

We invent here in this manuscript new tree describing parameters which can be derived from a QSM. QSMs are topological ordered cylinder models of trees which describe the branching structure completely. All new invented parameters have in common, that their defining point of view looks from the direction of the tips and not from the root along the tree. The reason here is simple, diameter relations are stronger when they rely on a distance measure to the tip(s) rather than on a distance measure to the root. In the traditional branch order for example diferent sized branches are contained, but in the reverse branch order this problem is barely exiting anymore. The pipe model theory (PMT) is a theory adapted to deciduous trees, based on the allometric scaling theory. And according to the PMT the count of new growth units can serve as a proxy to predict the sapwood area at the query cylinders cross section. By multiplying this area proxy with the cylinder length we receive a proxy for the sapwood volume contained. We name the sapwood volume of the whole subbranch VesselVolume. The sapwood volume finally serves as the predictor for the basic allometry function to predict the diameter/radius of combined sap and heartwood at the query point. For validation we use QSMs produced from an open point cloud data set of tree clouds with SimpleForest software. We compare the QSM volume against the harvested reference data for 66 felled trees. We also found QSM data of TreeQSM, a competitive and broadly accepted QSM modeling tool. Our RMSE was less than 40 % of the TreeQSM RMSE. And for other error measures, the r2 adj. and the CCC, the relative improvement looked even better with reaching only 27 % and 21 % of the TreeQSM errors respectively. With the invention of this filter we improve tree volume prediction capabilities utilizing QSMs. Additionally, we run numerical tests against the West Brown Enquist (WBE) model predictions using our filtered QSMs.

show abstract

Vis-MVSNet: Visibility-Aware Multi-view Stereo Network

Cited by 83 publications

References 51 publications

Enhancing Multi-View Stereo with Contrastive Matching and Weighted Focal Loss

Enhancing Multi-View Stereo with Contrastive Matching and Weighted Focal Loss

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

Improving quantitative structure models of trees inspired by pipe and metabolic scaling theory

Contact Info

Product

Resources

About