Baking Neural Radiance Fields for Real-Time View Synthesis

Abstract

Neural volumetric representations such as Neural Radiance Fields (NeRF) have emerged as a compelling technique for learning to represent 3D scenes from images with the goal of rendering photorealistic images of the scene from unobserved viewpoints. However, NeRF's computational requirements are prohibitive for real-time applications: rendering views from a trained NeRF requires querying a multilayer perceptron (MLP) hundreds of times per ray. We present a method to train a NeRF, then precompute and store (i.e. "bake") it as a novel representation called a Sparse Neural Radiance Grid (SNeRG) that enables realtime rendering on commodity hardware. To achieve this, we introduce 1) a reformulation of NeRF's architecture, and 2) a sparse voxel grid representation with learned feature vectors. The resulting scene representation retains NeRF's ability to render fine geometric details and view-dependent appearance, is compact (averaging less than 90 MB per scene), and can be rendered in real-time (higher than 30 frames per second on a laptop GPU). Actual screen captures are shown in our video.

Point-NeRF: Point-based Neural Radiance Fields

“…Voxel grids with per-voxel neural features [8,16,29] are also a local neural radiance representation. However, our point-based representation adapts better to actual surfaces, leading to better quality.…”

“…However, our point-based representation adapts better to actual surfaces, leading to better quality. Also, we directly predict good initial neural point features, bypassing the per-scene optimization that is required by most voxel-based methods [16,29].…”

“…They have been extended to achieve dynamic scene capture [27,39], relighting [3,5], appearance editing [57], fast rendering [16,61], and generative models [7,36,45]. However, most methods [3,27,39,57] still follow the original NeRF framework and train per-scene MLPs to represent radiance fields.…”

CG-NeRF: Conditional Generative Neural Radiance Fields

“…Neural Radiance Fields Recent advancements (Mildenhall et al 2020;Hedman et al 2021;Jain, Tancik, and Abbeel 2021;Srinivasan et al 2021;Yu et al 2021;Lindell, Martel, and Wetzstein 2021) in the area of novel view synthesis have been accomplished by employing the NeRF. The seminal work (Mildenhall et al 2020) has proven the effectiveness of volume rendering with NeRF, and later studies (Hedman et al 2021;Wang et al 2021;Zhang et al 2020) proposed further improvements over the original NeRF. While some NeRF studies enhance the original NeRF in terms of both quality and efficiency, our work is more related to generative NeRF methods, which have attracted attention recently.…”

Representing 3D Shapes with Probabilistic Directed Distance Fields

“…Neural Radiance Fields NeRFs [44] are powerful 3D representations, capable of novel view synthesis for reconstruction [83] and image generation [3] with very high fidelity. However, the standard differentiable volume render- ing formulation of NeRFs is computationally expensive, requiring many forward passes per pixel, though recent work has improved on this (e.g., [2,10,17,27,35,56,57,82]). Furthermore, the distributed nature of the density makes extracting explicit geometric details (including higher-order surface information) more difficult (e.g., [50,81]).…”