Hallucinated Neural Radiance Fields in the Wild

Abstract

Neural Radiance Fields (NeRF) have been proposed for photorealistic novel view rendering. However, it requires many different views of one scene for training. Moreover, it has poor generalizations to new scenes and requires retraining or finetuning on each scene. In this paper, we develop a new NeRF model for novel view synthesis using only a single image as input. We propose to combine the (coarse) planar rendering and the (fine) volume rendering to achieve higher rendering quality and better generalizations. We also design a depth teacher net that predicts dense pseudo depth maps to supervise the joint rendering mechanism and boost the learning of consistent 3D geometry. We evaluate our method on three challenging datasets. It outperforms state-of-the-art single-view NeRFs by achieving 5∼20% improvements in PSNR and reducing 20∼50% of the errors in the depth rendering. It also shows excellent generalization abilities to unseen data without the need to finetune on each new scene.

Advances in Neural Rendering

“…They apply their robust model to the Photo‐Tourism dataset [SSS06] (consisting of internet images of famous landmarks across the world) and are able to remove transient objects such as people and cars and capture time‐varying appearance through use of a latent code embedding associated with each input image. Ha‐NeRF [CZL*21] extends the idea of NeRF in the Wild to hallucinate novel appearances.…”

Enhancing image resolution of confocal fluorescence microscopy with deep learning

“…The method converts point clouds into a set of frequency-modulated signals that can be rendered efficiently using Fourier analysis. The signals can also be manipulated in the frequency domain to achieve various editing effects, such as deformation, smoothing, sharpening, and color adjustment.Chen et al also proposed NeuralEditor, a novel method for editing neural radiance fields (NeRFs) for shape editing tasks [156]. The method uses point clouds as the underlying structure to construct NeRFs and renders them with a new scheme based on K-D tree-guided voxels.…”

NeRF for Outdoor Scene Relighting

“…Style-based Editing. Scene relighting techniques are distantly related to style-based category of appearance editing methods [4,15,17,19,22,24,31,37]. Unlike relighting methods, the latter do not have a physical understanding of the scene illumination and seek to edit the overall appearance at once.…”

“…An exception to this is, at first sight, NeRF in the Wild (NeRF-W) [19] trained from uncontrolled images, factoring per-image appearance into an embedding space. However, NeRF-W and more recent follow-ups [4,37] do not perform intrinsic image decomposition and thus semantically meaningful parametric control of lighting, shadows or even albedo is not possible.…”