Coherent intrinsic images from photo collections

Abstract

Figure 1: Our method leverages the heterogeneity of photo collections to automatically decompose photographs of a scene into reflectance and illumination layers. The extracted reflectance layers are coherent across all views, while the illumination captures the shading and shadow variations proper to each picture. Here we show the decomposition of three photos in the collection. AbstractAn intrinsic image is a decomposition of a photo into an illumination layer and a reflectance layer, which enables powerful editing such as the alteration of an object's material independently of its illumination. However, decomposing a single photo is highly under-constrained and existing methods require user assistance or handle only simple scenes. In this paper, we compute intrinsic decompositions using several images of the same scene under different viewpoints and lighting conditions. We use multi-view stereo to automatically reconstruct 3D points and normals from which we derive relationships between reflectance values at different locations, across multiple views and consequently different lighting conditions. We use robust estimation to reliably identify reflectance ratios between pairs of points. From these, we infer constraints for our optimization and enforce a coherent solution across multiple views and illuminations. Our results demonstrate that this constrained optimization yields high-quality and coherent intrinsic decompositions of complex scenes. We illustrate how these decompositions can be used for image-based illumination transfer and transitions between views with consistent lighting.

Lighting transfer across multiple views through local color transforms

“…We also show time-lapse sequences synthesized by transferring all illumination conditions to a single viewpoint. In addition, we show a side-by-side comparison with the results of Laffont et al [14]. We also include relighting results where a person is present in the landmark photos and occupies a significant part of the scene.…”

“…These sophisticated systems make use of detailed geometric models and require registration or non-linear fitting. Laffont et al [14] show that intrinsic image decomposition can be used for illumination transfer, but the extraction of consistent reflectance and illumination layers is a challenging and computationally expensive problem. Alternatively, some methods transfer an image by learning color changes from correspondences of image pairs.…”

“…Pitie et al's method [2] produces a more similar tone to the target, but both of them do not properly transfer local lighting. In the result of Laffont et al's method [14], which uses intrinsic image decomposition, the regions in shadow are washed-out, and there are artifacts around the boundaries of the sky and buildings. In contrast, our result has lighting similar to that of the target image, and people and objects in shadow still have their colors.…”

“…We refer to the supplementary video in Fig. 7 We compare the global color transfer method [3], intrinsic image decomposition [14], and our lighting transfer method. While the results of other methods have either wrong color tone (c) or artifacts (d), our result has appropriate lighting similar to the target image (b).…”

White Balancing by Using Multiple Images via Intrinsic Image Decomposition

“…The reflectance component is a specific object color that does not depend on scene illumination unlike the shading component, which is a shading effect caused by the illumination lights. Based on their theory, many intrinsic image decomposition techniques have been proposed by a number of authors [9]- [17], and it is useful for several applications in computer vision, such as scene illumination transfer [15] and colorization [17]. However, intrinsic image decomposition is a major challenge due to the ill-posedness that there are two unknown components for one observed image.…”