Long-range Attention Network for Multi-View Stereo

Zhang, Xudong; Hu, Yutao; Wang, Haochen; Cao, Xianbin; Zhang, Baochang

doi:10.1109/wacv48630.2021.00383

Cited by 25 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comp. Overall Gipuma [8] 0.283 0.873 0.578 COLMAP [24], [25] 0.400 0.664 0.532 MVSNet [36] 0.396 0.527 0.462 R-MVSNet [37] 0.383 0.452 0.417 CasMVSNet [10] 0.346 0.351 0.348 PatchmatchNet [29] 0.427 0.277 0.352 D 2 HC-RMVSNET [34] 0.395 0.378 0.386 EPP-MVSNet [20] 0.413 0.296 0.355 AA-RMVSNet [31] 0.376 0.339 0.357 AACVP-MVSNet [39] 0.357 0.326 0.341 AttMVS [19] 0.383 0.329 0.356 LANet [41] 0.320 0.349 0.335 Ours 0.278 0.377 0.327 be found in Table II, where our approach ranks amongst the top published methods, while keeping runtime and GPU memory requirements low. Furthermore we compare our method qualitatively to EPP-MVSNet [20], ranked highest in Table II, in Figure 6.…”

Section: Methodsmentioning

confidence: 99%

“…In the task of object detection and image classification, the attention mechanism has been successful in achieving gains by augmenting convolutional models with content-based interactions [3]. This motivated several works [19], [41] to capitalize on the new technique also in MVS. A first attempt to exploit the local attention layer proposed by [21] is performed by Yu et al [39].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

ATLAS-MVSNet: Attention Layers for Feature Extraction and Cost Volume Regularization in Multi-View Stereo

Weilharter

Fraundorfer

2022

2022 26th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

We present ATLAS-MVSNet, an end-to-end deep learning architecture relying on local attention layers for depth map inference from multi-view images. Distinct from existing works, we introduce a novel module design for neural networks, which we termed hybrid attention block, that utilizes the latest insights into attention in vision models. We are able to reap the benefits of attention in both, the carefully designed multi-stage feature extraction network and the cost volume regularization network. Our new approach displays significant improvement over its counterpart based purely on convolutions. While many state-of-the-art methods need multiple high-end GPUs in the training phase, we are able to train our network on a single consumer grade GPU. ATLAS-MVSNet exhibits excellent performance, especially in terms of accuracy, on the DTU dataset. Furthermore, ATLAS-MVSNet ranks amongst the top published methods on the online Tanks and Temples benchmark.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

ATLAS-MVSNet: Attention Layers for Feature Extraction and Cost Volume Regularization in Multi-View Stereo

Weilharter

Fraundorfer

2022

2022 26th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

show abstract

“…The attention [39][40][41] is also proven to be able to extract representative features from ambiguous regions. Finally, the fused light field features will go through another spatial-angular regularisation module to implicitly regularise the structure of LF contents:…”

Section: Fusementioning

confidence: 99%

Robust Local Light Field Synthesis via Occlusion-aware Sampling and Deep Visual Feature Fusion

2023

View full text Add to dashboard Cite

Novel view synthesis has attracted tremendous research attention recently for its applications in virtual reality and immersive telepresence. Rendering a locally immersive light field (LF) based on arbitrary large baseline RGB references is a challenging problem that lacks efficient solutions with existing novel view synthesis techniques. In this work, we aim at truthfully rendering local immersive novel views/LF images based on large baseline LF captures and a single RGB image in the target view. To fully explore the precious information from source LF captures, we propose a novel occlusion-aware source sampler (OSS) module which efficiently transfers the pixels of source views to the target view’s frustum in an occlusion-aware manner. An attention-based deep visual fusion module is proposed to fuse the revealed occluded background content with a preliminary LF into a final refined LF. The proposed source sampling and fusion mechanism not only helps to provide information for occluded regions from varying observation angles, but also proves to be able to effectively enhance the visual rendering quality. Experimental results show that our proposed method is able to render high-quality LF images/novel views with sparse RGB references and outperforms state-of-the-art LF rendering and novel view synthesis methods.

show abstract

“…CasMVSNet [19] adopts a cascade cost volume to gradually narrow the depth range and increase the cost volume resolution. Similar ideas are later explored to reduce the memory cost of 3D convolutions and/or increase the depth quality, such as coarse-to-fine depth optimization [10], [39], [68], [69], [71], [72], [79], attention-based feature aggregation [38], [66], [78], [84], and patch matching-based method [37], [62]. Unlike these works, RayMVSNet optimizes the depth on each camera viewing ray instead of the 3D volume, which is more light-weight.…”

Section: Related Workmentioning

confidence: 99%

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

Shi²,

Wang

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Learning-based multi-view stereo (MVS) has by far centered around 3D convolution on cost volumes. Due to the high computation and memory consumption of 3D CNN, the resolution of output depth is often considerably limited. Different from most existing works dedicated to adaptive refinement of cost volumes, we opt to directly optimize the depth value along each camera ray, mimicking the range (depth) finding of a laser scanner. This reduces the MVS problem to ray-based depth optimization which is much more light-weight than full cost volume optimization. In particular, we propose RayMVSNet which learns sequential prediction of a 1D implicit field along each camera ray with the zero-crossing point indicating scene depth. This sequential modeling, conducted based on transformer features, essentially learns the epipolar line search in traditional multi-view stereo. We devise a multi-task learning for better optimization convergence and depth accuracy. We found the monotonicity property of the SDFs along each ray greatly benefits the depth estimation. Our method ranks top on both the DTU and the Tanks & Temples datasets over all previous learning-based methods, achieving an overall reconstruction score of 0.33mm on DTU and an F-score of 59.48% on Tanks & Temples. It is able to produce high-quality depth estimation and point cloud reconstruction in challenging scenarios such as objects/scenes with non-textured surface, severe occlusion, and highly varying depth range. Further, we propose RayMVSNet++ to enhance contextual feature aggregation for each ray through designing an attentional gating unit to select semantically relevant neighboring rays within the local frustum around that ray. This improves the performance on datasets with more challenging examples (e.g. low-quality images caused by poor lighting conditions or motion blur). RayMVSNet++ achieves state-of-the-art performance on the ScanNet dataset. In particular, it attains an AbsRel of 0.058m and produces accurate results on the two subsets of textureless regions and large depth variation.

show abstract

Long-range Attention Network for Multi-View Stereo

Cited by 25 publications

References 25 publications

ATLAS-MVSNet: Attention Layers for Feature Extraction and Cost Volume Regularization in Multi-View Stereo

ATLAS-MVSNet: Attention Layers for Feature Extraction and Cost Volume Regularization in Multi-View Stereo

Robust Local Light Field Synthesis via Occlusion-aware Sampling and Deep Visual Feature Fusion

RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo

Contact Info

Product

Resources

About