A signal-processing framework for inverse rendering

Abstract

Realism in computer-generated images requires accurate input models for lighting, textures and BRDFs. One of the best ways of obtaining high-quality data is through measurements of scene attributes from real photographs by inverse rendering. However, inverse rendering methods have been largely limited to settings with highly controlled lighting. One of the reasons for this is the lack of a coherent mathematical framework for inverse rendering under general illumination conditions. Our main contribution is the introduction of a signal-processing framework which describes the reflected light field as a convolution of the lighting and BRDF, and expresses it mathematically as a product of spherical harmonic coefficients of the BRDF and the lighting. Inverse rendering can then be viewed as deconvolution. We apply this theory to a variety of problems in inverse rendering, explaining a number of previous empirical results. We will show why certain problems are ill-posed or numerically ill-conditioned, and why other problems are more amenable to solution. The theory developed here also leads to new practical representations and algorithms. For instance, we present a method to factor the lighting and BRDF from a small number of views, i.e. to estimate both simultaneously when neither is known.

Recovery of Reflectances and Varying Illuminants from Multiple Views

“…It is, however, not generic because it depends on a specific situation where the shadow of an object of known shape is projected onto an object also of known shape. Recent work [18,19] done simultaneously to ours [12] demonstrated that a complex lighting model, including multiple lightsources and specular properties, can be recovered given several images of an object for which a 3-D model is available but, unlike in our approach, requires assuming that the lighting is the same in all images.…”

Inverse Rendering of Faces with a 3D Morphable Model

“…For example, Ramamoorthi et al [72] point out that it is not possible to distinguish between low-frequency texture and lighting effects. They suggest that this ambiguity can only be resolved by using active methods or making assumptions about the expected characteristics of the texture and lighting.…”

“…Ramamoorthi and Hanrahan [72] used spherical harmonics to describe the reflected light field as a convolution of lighting and reflectance. They present a signal processing framework for a variety of inverse rendering problems under the assumption of known geometry.…”

A Dual Theory of Inverse and Forward Light Transport

“…Previous methods have considered inverting the direct reflection equation to acquire lighting and reflectance properties [Marschner 1998;Sato et al 1999;Ramamoorthi and Hanrahan 2001]. [Yu et al 1999] develop an inverse global illumination method for BRDF estimation.…”

“…2 Therefore, we consider projector-based acquisition, that illuminates a single spatial location and records the response, rather than light sources that illuminate the whole object (where F is a low-pass filter, that is not easy to invert for diffuse surfaces [Ramamoorthi and Hanrahan 2001]). For projectorbased acquisition, after geometric calibration, we can use the same parameterization for projection and camera images [Seitz et al 2005].…”