RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

Chang, Di; Božič, Aljaž; Zhang, Tong; Yan, Qingsong; Chen, Ying-Cong; Süsstrunk, Sabine; Nießner, Matthias

doi:10.1007/978-3-031-19821-2_38

Cited by 24 publications

(20 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to verify the effectiveness of the model, this paper visualizes the results of the open-source models MVSNet [4] and RC-MVSNet [7] , and compares the 3D reconstruction effect of this paper's model and other models in typical scenes of the DTU dataset, and the results are shown in Fig. 2.…”

Section: Results On Dtu Datasetmentioning

confidence: 99%

“…Comp. Overall Sup SurfaceNet [3] 0.450 1.040 0.745 MVSNet [4] 0.396 0.527 0.462 R-MVSNet [3] 0.389 0.455 0.382 Cas-MVSNet [3] 0.325 0.385 0.355 CVP-MVSNet [3] 0.296 0.406 0.351 Unsup Unsup-MVSNet [3] MVS2 [3] JDACS [3] 0.881 0.760 0.398 1.073 0.515 0.318 0.977 0.637 0.358 RC-MVSNet [7] 0.396 0.295 0.345 TRC-MVSNet 0.366 0.297 0.332…”

Section: Results On Dtu Datasetmentioning

confidence: 99%

“…In the field of multi-view 3D reconstruction, structure from motion (SFM) is a technique for estimating 3D structures from a series of multiple 2D image sequences containing visual motion information [1]- [2] , and the Multi-view stereo (MVS) method [3]- [4] , which combines multi-view stereo vision and deep learning [5]- [7] , has become one of the research hotspots in the field of 3D reconstruction. However, the methods based on multi-view infrared imaging also face challenges in improving the accuracy of infrared image matching and fusion.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Infrared target 3D reconstruction based on TRC-MVSNet

Gong,

Jin,

Tang

et al. 2024

International Conference on Mechatronic Engineering and Artificial Intelligence (MEAI 2023)

View full text Add to dashboard Cite

A 3D reconstruction network, TRC-MVSNet, is proposed to address the issue of poor 3D reconstruction of infrared multi-view. It combines Transformer feature matching and RC-MVSNet. To enhance the quality of the infrared image, a moving average filter method is employed to mitigate noise by averaging signals in a sliding window. Additionally, the Transformer network is utilized to improve the matching quality of infrared multi-view information by matching the information of multiple infrared views in the feature matching stage. Experimental results demonstrate that the TRC-MVSNet 3D reconstruction model enhances accuracy by 0.03 and synthesis by 0.013 compared to the RC-MVSNet 3D reconstruction model on the DTU dataset. The TRC-MVSNet achieves high accuracy, efficiency, and minimal noise in the 3D reconstruction of infrared targets on the SYLU-IMD dataset, outperforming other methods.

show abstract

Section: Results On Dtu Datasetmentioning

confidence: 99%

Section: Results On Dtu Datasetmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Infrared target 3D reconstruction based on TRC-MVSNet

Gong,

Jin,

Tang

et al. 2024

International Conference on Mechatronic Engineering and Artificial Intelligence (MEAI 2023)

View full text Add to dashboard Cite

show abstract

“…Our depth estimation network uses the same architecture as proposed by Deschaintre et al [DLG21], with a novel depth‐based rendering loss. Unlike the depth rendering loss recently proposed by Chang et al [CBZ*22] which uses a NERF style volumetric depth rendering, we solely utilize the predicted depth as the input for rendering and subsequently transform this predicted depth into surface normals. We designate specific parameters of the BRDF, such as roughness, to constant values in accordance with the conventional Cook‐Torrance model and calculate the rendering results loss between the ground truth rendering.…”

Section: Methodsmentioning

confidence: 99%

Deep Shape and SVBRDF Estimation using Smartphone Multi‐lens Imaging

Fan,

Lin,

Ghosh

2023

Computer Graphics Forum

View full text Add to dashboard Cite

We present a deep neural network‐based method that acquires high‐quality shape and spatially varying reflectance of 3D objects using smartphone multi‐lens imaging. Our method acquires two images simultaneously using a zoom lens and a wide angle lens of a smartphone under either natural illumination or phone flash conditions, effectively functioning like a single‐shot method. Unlike traditional multi‐view stereo methods which require sufficient differences in viewpoint and only estimate depth at a certain coarse scale, our method estimates fine‐scale depth by utilising an optical‐flow field extracted from subtle baseline and perspective due to different optics in the two images captured simultaneously. We further guide the SVBRDF estimation using the estimated depth, resulting in superior results compared to existing single‐shot methods.

show abstract

“…Early data-driven MVS techniques include SurfaceNet [21,22], which combines all pixel and camera information into colored voxel cubes used as input of the network, and LSM [18], which uses differentiable projection for end-to-end training. MVSNet [23,24] infers the depth maps for each image one at a time using multiple source images and uses differentiable homography warping to enable end-to-end training. These methods have leveraged DNNs to obtain high-quality reconstructions, and recent works, such as R-MVSNet [25], put the focus on scalability.…”

Section: Related Workmentioning

confidence: 99%

Iterative Online 3D Reconstruction from RGB Images

Cardoen

Leroux

Simoens

2022

Sensors

View full text Add to dashboard Cite

3D reconstruction is the computer vision task of reconstructing the 3D shape of an object from multiple 2D images. Most existing algorithms for this task are designed for offline settings, producing a single reconstruction from a batch of images taken from diverse viewpoints. Alongside reconstruction accuracy, additional considerations arise when 3D reconstructions are used in real-time processing pipelines for applications such as robot navigation or manipulation. In these cases, an accurate 3D reconstruction is already required while the data gathering is still in progress. In this paper, we demonstrate how existing batch-based reconstruction algorithms lead to suboptimal reconstruction quality when used for online, iterative 3D reconstruction and propose appropriate modifications to the existing Pix2Vox++ architecture. When additional viewpoints become available at a high rate, e.g., from a camera mounted on a drone, selecting the most informative viewpoints is important in order to mitigate long term memory loss and to reduce the computational footprint. We present qualitative and quantitative results on the optimal selection of viewpoints and show that state-of-the-art reconstruction quality is already obtained with elementary selection algorithms.

show abstract

RC-MVSNet: Unsupervised Multi-View Stereo with Neural Rendering

Cited by 24 publications

References 33 publications

Infrared target 3D reconstruction based on TRC-MVSNet

Infrared target 3D reconstruction based on TRC-MVSNet

Deep Shape and SVBRDF Estimation using Smartphone Multi‐lens Imaging

Iterative Online 3D Reconstruction from RGB Images

Contact Info

Product

Resources

About