DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

Abstract

The recent research explosion around implicit neural representations, such as NeRF, shows that there is immense potential for implicitly storing high‐quality scene and lighting information in compact neural networks. However, one major limitation preventing the use of NeRF in real‐time rendering applications is the prohibitive computational cost of excessive network evaluations along each view ray, requiring dozens of petaFLOPS. In this work, we bring compact neural representations closer to practical rendering of synthetic content in real‐time applications, such as games and virtual reality. We show that the number of samples required for each view ray can be significantly reduced when samples are placed around surfaces in the scene without compromising image quality. To this end, we propose a depth oracle network that predicts ray sample locations for each view ray with a single network evaluation. We show that using a classification network around logarithmically discretized and spherically warped depth values is essential to encode surface locations rather than directly estimating depth. The combination of these techniques leads to DONeRF, our compact dual network design with a depth oracle network as its first step and a locally sampled shading network for ray accumulation. With DONeRF, we reduce the inference costs by up to 48× compared to NeRF when conditioning on available ground truth depth information. Compared to concurrent acceleration methods for raymarching‐based neural representations, DONeRF does not require additional memory for explicit caching or acceleration structures, and can render interactively (20 frames per second) on a single GPU.

Plenoxels: Radiance Fields without Neural Networks

“…Learned Initializations [49] employs meta-learning on many scenes to start from a better MLP initialization, for both > 10× faster training and better priors when per-scene data is limited. Other methods [14,27,33] achieve speedup by predicting a surface or sampling near the surface, reducing the number of samples necessary for rendering each ray.…”

Artemis: Articulated Neural Pets with Appearance and Motion synthesis

“…Recently NeRF ] has achieved impressive progress by learning radiance fields from 2D images. The following variants of NeRF aim to learn generalizable radiance fields , train with unposed cameras [Meng et al 2021;Yen-Chen et al 2020], model highfrequency details [Luo et al 2021b;Sitzmann et al 2020; and opacity [Luo et al 2021b] or accelerate for real-time applications [Garbin et al 2021;Neff et al 2021;Reiser et al 2021;. Most recent approaches extend NeRF to dynamic scenes by learning a deformation field Park et al 2020;Pumarola et al 2021;Tretschk et al 2021;Xian et al 2021] or training a hypernet [Park et al 2021].…”

NeROIC: Neural Rendering of Objects from Online Image Collections

“…iNeRF [44] recovers ground-truth poses by inverting a trained neural radiance field to render the input images. Other works focus on improving the training or inference performance and computational efficiency [17,18,23,31,39,45]). Related approaches [1,10] use a signed-distance function to represent a surface that can be extracted as a mesh for fast rendering and novel view synthesis.…”