Illuminant Spectra-Based Source Separation Using Flash Photography

Abstract

Real-world lighting often consists of multiple illuminants with different spectra. Separating and manipulating these illuminants in post-process is a challenging problem that requires either significant manual input or calibrated scene geometry and lighting. In this work, we leverage a flash/noflash image pair to analyze and edit scene illuminants based on their spectral differences. We derive a novel physicsbased relationship between color variations in the observed flash/no-flash intensities and the spectra and surface shading corresponding to individual scene illuminants. Our technique uses this constraint to automatically separate an image into constituent images lit by each illuminant. This separation can be used to support applications like white balancing, lighting editing, and RGB photometric stereo, where we demonstrate results that outperform state-of-theart techniques on a wide range of images. We introduce a novel reflectance-invariant property ofLambertian scenes that relates illuminant spectra to observed pixel intensities.2. We propose an algorithm to separate an image into its constituent illuminants, and present an analysis of the robustness and limitations of this technique.3. We leverage these separated images to enable a wide variety of applications including white balancing, light editing, camera response editing and photometric stereo.Assumptions. The techniques proposed in this paper assume that the imaged scene is Lambertian and that scene

Learning to Separate Multiple Illuminants in a Single Image

“…Once trained, we find that our approach is able to successfully solve this ill-posed problem and produce high-quality lighting decompositions that, as can be seen in Figure 1, capture complex shading and shadows. In fact, our network is able to match, and in specific instances outperform, the quality of results from Hui et al's two-image method [25], despite needing only a single image as input.…”

“…Thus, past efforts to achieve such separation have relied heavily on extensive manual annotation [8,7,9] or access to calibrated scene and lighting information [12,11]. More recently, Hui et al [24,25] demonstrate that the lighting separation problem can be reliably solved if one additionally knows the reflectance chromaticity of all surface (a) Input image (b) Output separated images Figure 1. Our method separates a single image (a) captured under two illuminants with different spectra (sun and sky illumination here) into two images corresponding to the appearance of the scene under the individual lights.…”

“…However, we find that standard architectures, trained only with the respect to the quality of the final separated images, are unable to learn to effectively perform the separation. Therefore, we guide the design of our network using a physics-based analysis of the task [25] to match the expected sequence of inference steps and intermediate outputs -reflectance chromaticities, shading chromaticities, separated shading maps, and final separated images. In addition to ensuring that our architecture has the ability to express these required computations, this decomposition also allows us to provide supervision to intermediate layers in our network, which proves crucial to successful training.…”

“…c c i = 1. Similar to Hui et al [25], we assume that the light source chromaticities are unique, i.e., 1 = 2 . The term λ i (p) is the shading observed at pixel p due to the…”

A Dataset of Multi-Illumination Images in the Wild

“…from the room's artificial tungsten light bulbs). Prior work has formulated a local gray-world assumption to generalize white balance to the mixed-lighting case [11], exploiting the difference in light colors in shadowed vs. sunlit areas for outdoor scenes [25], or flash/no-flash image pairs [33,21,23].…”