RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

Niemeyer, Michael; Barron, Jonathan T.; Mildenhall, Ben; Sajjadi, Mehdi S. M.; Geiger, Andreas; Radwan, Noha

doi:10.1109/cvpr52688.2022.00540

Cited by 191 publications

(94 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural Radiance Fields (NeRF) [24] propose an effective implicit neural scene representation method to synthesize novel views by the single scene optimization. Many subsequent coordinate-based methods are inspired by it, and we can classify them into neural surface fields [27,42,37] and neural radiance fields [26,16,44] based on the shape representation differences. However, both ways have poor performance on large scenes due to the limited representation capability of MLP-based networks adopted in NeRF.…”

Section: Neural Scene Representation and Renderingmentioning

confidence: 99%

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction

Lu¹,

Yin²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Neural Radiance Fields (NeRF) has achieved impressive results in single object scene reconstruction and novel view synthesis, as demonstrated on many single modality and single object focused indoor scene datasets like DTU [14], BMVS [41], and NeRF Synthetic [24]. However, the study of NeRF on large-scale outdoor scene reconstruction is still limited, as there is no unified outdoor scene dataset for large-scale NeRF evaluation due to expensive data acquisition and calibration costs.In this work, we propose a large-scale outdoor multimodal dataset, OMMO dataset, containing complex objects and scenes with calibrated images, point clouds and prompt annotations. A new benchmark for several outdoor NeRF-based tasks is established, such as novel view synthesis, surface reconstruction, and multi-modal NeRF. To create the dataset, we capture and collect a large number of real fly-view videos and select high-quality and highresolution clips from them. Then we design a quality review module to refine images, remove low-quality frames and fail-to-calibrate scenes through a learning-based automatic evaluation plus manual review. Finally, a number of volunteers are employed to add the text descriptions for each scene and keyframe.Compared with existing NeRF datasets, our dataset contains abundant real-world urban and natural scenes with various scales, camera trajectories, and lighting conditions. Experiments show that our dataset can benchmark most state-of-the-art NeRF methods on different tasks. We will release the dataset and model weights soon.

show abstract

Section: Neural Scene Representation and Renderingmentioning

confidence: 99%

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction

Lu¹,

Yin²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, it mainly considers the enhancement of the density aspect. RegNeRF [73] proposes to add constraints on both density and color to reduce the number of input images required, where the color constraint is patch-based supervision. Our method also introduces depth and patch-based supervision, but we propose a pre-training process using mesh rendering as pseudo ground truth and introduce the 3D voxel color prior, which provides more prior knowledge.…”

Section: Related Workmentioning

confidence: 99%

Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

Yuan

Lai

Huang

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

The recently proposed neural radiance fields (NeRF) use a continuous function formulated as a multi-layer perceptron (MLP) to model the appearance and geometry of a 3D scene. This enables realistic synthesis of novel views, even for scenes with view dependent appearance. Many follow-up works have since extended NeRFs in different ways. However, a fundamental restriction of the method remains that it requires a large number of images captured from densely placed viewpoints for high-quality synthesis and the quality of the results quickly degrades when the number of captured views is insufficient. To address this problem, we propose a novel NeRF-based framework capable of high-quality view synthesis using only a sparse set of RGB-D images, which can be easily captured using cameras and LiDAR sensors on current consumer devices. First, a geometric proxy of the scene is reconstructed from the captured RGB-D images. Renderings of the reconstructed scene along with precise camera parameters can then be used to pre-train a network. Finally, the network is fine-tuned with a small number of real captured images. We further introduce a patch discriminator to supervise the network under novel views during fine-tuning, as well as a 3D color prior to improve synthesis quality. We demonstrate that our method can generate arbitrary novel views of a 3D scene from as few as 6 RGB-D images. Extensive experiments show the improvements of our method compared with the existing NeRF-based methods, including approaches that also aim to reduce the number of input images.

show abstract

“…To mitigate this problem, we take advantage of the geometry regularization used in [36] and encourage the generated depth to be smooth, even from unobserved viewpoints. The regularization is based on the real-world observation that real geometry or depth tends to be smooth, and is more likely to be flat, and formulated such that depth for each pixel should not be too different from those of neighboring pixels.…”

Section: Loss Functionsmentioning

confidence: 99%

“…where H and W are the height and width of the generated depth map, and r i,j indicates the ray through pixel (i, j). Note that while [36] implements the geometry regularization by comparing overlapping patches, we utilize the full generated depth map D for our implementation.…”

Section: Loss Functionsmentioning

confidence: 99%

3D GAN Inversion with Pose Optimization

Ko¹,

Cho²,

Choi³

et al. 2022

Preprint

View full text Add to dashboard Cite

With the recent advances in NeRF-based 3D aware GANs quality, projecting an image into the latent space of these 3D-aware GANs has a natural advantage over 2D GAN inversion: not only does it allow multi-view consistent editing of the projected image, but it also enables 3D reconstruction and novel view synthesis when given only a single image. However, the explicit viewpoint control acts as a main hindrance in the 3D GAN inversion process, as both camera pose and latent code have to be optimized simultaneously to reconstruct the given image. Most works that explore the latent space of the 3D-aware GANs rely on ground-truth camera viewpoint or deformable 3D model, thus limiting their applicability. In this work, we introduce a generalizable 3D GAN inversion method that infers camera viewpoint and latent code simultaneously to enable multi-view consistent semantic image editing. The key to our approach is to leverage pre-trained estimators for better initialization and utilize the pixel-wise depth calculated from NeRF parameters to better reconstruct the given image.We conduct extensive experiments on image reconstruction and editing both quantitatively and qualitatively, and further compare our results with 2D GAN-based editing to demonstrate the advantages of utilizing the latent space of 3D GANs. Additional results and visualizations are available at 3dgan-inversion. github.io

show abstract

RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs

Cited by 191 publications

References 57 publications

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction

A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction

Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

3D GAN Inversion with Pose Optimization

Contact Info

Product

Resources

About