Rich Intrinsic Image Decomposition of Outdoor Scenes from Multiple Views

Abstract

In this appendix, we describe the details of the illuminant calibration step for the sky and the sun.First, because our model separates sun light from sky light, we need to remove sun pixels from the environment map. We define the sun position as the barycenter of the saturated sun pixels, and use inpainting to fill-in these saturated pixels from their neighbors. Since our model also separates sky light from indirect light, we use a standard color selection tool to label sky pixels that will contribute to the sky illumination, while other pixels (building, trees) will contribute to indirect lighting. This is illustrated in Fig. 5c. Second, we align the environment map and sun with the reconstructed scene. To do so we manually mark a vertical edge of the reconstructed geometry and rotate the environment map and sun until the cast shadow of the virtual edge is aligned with that in the photograph.Finally, the environment map only captures a scaled version of the scene lighting since the sphere is not perfectly specular. We need to compensate for this scaling factor in each color channel.In our system the environment map is used to compute both the sky illumination S sky and part of the indirect illumination S ind , the other part being computed from the geometric proxy (see Section 5 for more details). We estimate the color transfer function of the reflective sphere K (represented as a RGB vector) by taking a photograph of a neutral gray card with known reflectance R placed in sun shadow. We intentionally place the card at a position where we expect its geometry to be well reconstructed. From the image formation model we havewhere S env .denotes the illumination terms computed from the environment map and S proxy .the ones computed from the geometric proxy. We use this equation to solve for the unknown K.• REVES / INRIA Sophia AntipolisWe similarly recover the sun radiance L sun by taking a second picture of the gray card placed in sunlight (v sun = 1). From this picture we havewhere L sun is the only unknown.We could envisage other approaches for this calibration. Cabral et al.[1] use a compass to compute the environment map orientation with respect to the view direction in each photograph. In the work of Yu and Malik [2], a set of photographs of the horizon are taken, and a sky model is fitted. This involves taking several photographs, as opposed to the only two we take of the sphere, and also requires the use of two neutral lens filters to estimate sun radiance. In addition, fitting parameters of sky models can require specific tuning for relighting applications, as was the case for morning and evening sky in the work of Cabral et al. [1]. SUPPLEMENTARY VIDEOA supplementary video is attached. It illustrates some applications of our decomposition in image editing software. SUPPLEMENTARY FIGURESIn this document we show results on four scenes, for a total of 14 views. For each viewpoint, we provide the input (rectified) image, estimated reflectance, as well as the constraints and estimated total illumination, sun i...

Interactive intrinsic video editing

“…The quality of the decompositions can be improved by imposing priors on non-local reflectance [Shen et al 2008;Gehler et al 2011;Zhao et al 2012;Bell et al 2014], on gradient distributions [Li and Brown 2014;Shen and Yeo 2011], on the number of reflectance colors [Shen and Yeo 2011], or on the underlying scene geometry and illumination [Barron and Malik 2012]. Alternatively, the conditioning of the problem can be improved by leveraging multiple images captured under varying illumination [Weiss 2001;Matsushita et al 2004;Laffont et al 2012] or view directions [Laffont et al 2013], or by making use of depth information captured with RGBD cameras [Lee et al 2012;Barron and Malik 2013;Chen and Koltun 2013].…”

Learning to Remove Soft Shadows

“…For instance, Sinha and Adelson [1993] showed how to differentiate reflectance from illumination discontinuities in the world of painted polyhedra, which improved on previous approaches based on the Retinex theory [Land and McCann 1971]. Work by Laffont et al [2013] used a multi-view stereo reconstruction to estimate intrinsic images by combining 3D information with image propagation methods, while [Weiss 2001] used multiple images of the same object under varying lighting conditions and a prior based on statistics of natural images to obtain convincing reflectance and shading separation.…”

Parametric meta-filter modeling from a single example pair

“…When the filter is unknown, the result is a blind deconvolution problem. These techniques use some priors and regularization to constrain the solution and restrict the search space [5,9,10,[15][16][17][18][19]30]. Most filter estimation methods assume that a homogenous filter is applied to the whole image (or a sufficiently large region).…”