Learning Implicit Surface Light Fields

Abstract

Implicit representations of 3D objects have recently achieved impressive results on learning-based 3D reconstruction tasks. While existing works use simple texture models to represent object appearance, photo-realistic image synthesis requires reasoning about the complex interplay of light, geometry and surface properties. In this work, we propose a novel implicit representation for capturing the visual appearance of an object in terms of its surface light field. In contrast to existing representations, our implicit model represents surface light fields in a continuous fashion and independent of the geometry. Moreover, we condition the surface light field with respect to the location and color of a small light source. Compared to traditional surface light field models, this allows us to manipulate the light source and relight the object using environment maps. We further demonstrate the capabilities of our model to predict the visual appearance of an unseen object from a single real RGB image and corresponding 3D shape information. As evidenced by our experiments, our model is able to infer rich visual appearance including shadows and specular reflections. Finally, we show that the proposed representation can be embedded into a variational auto-encoder for generating novel appearances that conform to the specified illumination conditions.

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

“…Datasets. We evaluate our method on two public datasets: Photoshapes [28,36] with 150K chairs rendered at 128×128 following the rendering protocol of [25] and Carla with 10K cars rendered at 256×256 using the Driving simulator [6,36]. Each object is rendered in a random view without providing any camera pose parameters.…”

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

“…Neural Implicit Representations: Recently, neural implicit functions have emerged as an effective representation of 3D geometry [3,8,12,32,33,37,41,43,47,57] and appearance [23,25,34,38,39,40,47,49,52] as they represent 3D content continuously and without discretization while simultaneously having a small memory footprint. Most of these methods require 3D supervision.…”

“…Many recent methods employ continuous implicit functions parameterized with neural networks as 3D representations of geometry [3,8,12,32,33,37,41,43,47,57] or appearance [34,38,39,40,47,52,61]. These neural 3D representations have shown impressive performance on geometry reconstruction and novel view synthesis from multi-view images.…”

“…More specifically, NeRF uses a neural network to map a 3D location x ∈ R 3 and a viewing direction d ∈ R 3 to a volume density σ θ (x) ∈ R + and a color value c θ (x, d) ∈ R 3 . Conditioning on the viewing direction d allows for modeling view-dependent effects such as specular reflections [34,40] and improves reconstruction quality in case the Lambertian assumption is violated [61]. Let o denote the location of the camera center.…”

Learning Neural Light Fields with Ray-Space Embedding Networks

“…Coordinate-based representations have emerged as a powerful tool for overcoming the limitations of traditional discrete representations (e.g., images, meshes, voxelized volumes). The core idea is to train an MLP to map an input coordinate to the desired target value such as pixel color [26,32,45,60], signed distance [5,33], occupancy [28], volume density [30], or semantic labels [63]. Like existing coordinate-based representation approaches, our method also learns the mapping from an input coordinate (ray) to a target scene property (color).…”