Volume Rendering of Neural Implicit Surfaces

Abstract

Neural volume rendering became increasingly popular recently due to its success in synthesizing novel views of a scene from a sparse set of input images. So far, the geometry learned by neural volume rendering techniques was modeled using a generic density function. Furthermore, the geometry itself was extracted using an arbitrary level set of the density function leading to a noisy, often low fidelity reconstruction. The goal of this paper is to improve geometry representation and reconstruction in neural volume rendering. We achieve that by modeling the volume density as a function of the geometry. This is in contrast to previous work modeling the geometry as a function of the volume density. In more detail, we define the volume density function as Laplace's cumulative distribution function (CDF) applied to a signed distance function (SDF) representation. This simple density representation has three benefits: (i) it provides a useful inductive bias to the geometry learned in the neural volume rendering process; (ii) it facilitates a bound on the opacity approximation error, leading to an accurate sampling of the viewing ray. Accurate sampling is important to provide a precise coupling of geometry and radiance; and (iii) it allows efficient unsupervised disentanglement of shape and appearance in volume rendering. Applying this new density representation to challenging scene multiview datasets produced high quality geometry reconstructions, outperforming relevant baselines. Furthermore, switching shape and appearance between scenes is possible due to the disentanglement of the two.Preprint. Under review.

NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

“…More recently, by leveraging an implicit representation in the form of a Neural Radiance Field, showed the ability to model complex geometries and illumination from images. There has since been a flurry of impressive work to further push the boundaries of these representations and allow modeling deformation (Pumarola et al, 2020;Park et al, 2021), lighting variation (Martin-Brualla et al, 2021), and similar to ours, leveraging insights from surface rendering to model radiance (Yariv et al, 2020;Boss et al, 2021;Oechsle et al, 2021;Srinivasan et al, 2021;Zhang et al, 2021b;Wang et al, 2021a;Yariv et al, 2021). However, unlike our approach which can efficiently learn from a sparse set of images with coarse cameras, these approaches rely on a dense set of multi-view images with precise camera localization to recover a coherent 3D structure of the scene.…”

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

“…Our solution to the first problem is to add a parsing branch. Since the portrait parts of the same semantic category share similar motion patterns and texture information, it will be beneficial for the appearance and geometry learning in NeRF, which is also proven in recent implicit representation studies [73][74][75]81]. As shown in Fig.…”

GeoNeRF: Generalizing NeRF with Geometry Priors

“…DS-NeRF [11] shows that depth supervision can help NeRF train faster with fewer input views. Moreover, numerous works [24,38,52,58] show that despite the highquality color rendering, NeRF has difficulty reconstructing 3D geometry and surface normals. Accordingly, for training samples coming from datasets with ground truth depths, we also output the predicted depth d for each ray and supervise it if the ground truth depth of that pixel is available:…”