Interactive intrinsic video editing

Abstract

. Now, editing the textures in the reflectance image does not affect the illumination (c): changes to the brick walls, the roof tiles, and the pathway leading up to the building all maintain the complex illumination of the light through the trees. We encourage readers to zoom into this figure to see the details in our results, and refer them to the accompanying video to see the temporally consistent nature of our decomposition. AbstractSeparating a photograph into its reflectance and illumination intrinsic images is a fundamentally ambiguous problem, and state-of-the-art algorithms combine sophisticated reflectance and illumination priors with user annotations to create plausible results. However, these algorithms cannot be easily extended to videos for two reasons: first, naïvely applying algorithms designed for single images to videos produce results that are temporally incoherent; second, effectively specifying user annotations for a video requires interactive feedback, and current approaches are orders of magnitudes too slow to support this. We introduce a fast and temporally consistent algorithm to decompose video sequences into their reflectance and illumination components. Our algorithm uses a hybrid 2-p formulation that separates image gradients into smooth illumination and sparse reflectance gradients using look-up tables. We use a multi-scale parallelized solver to reconstruct the reflectance and illumination from these gradients while enforcing spatial and temporal reflectance constraints and user annotations. We demonstrate that our algorithm automatically produces reasonable results, that can be interactively refined by users, at rates that are two orders of magnitude faster than existing tools, to produce high-quality decompositions for challenging real-world video sequences. We also show how these decompositions can be used for a number of video editing applications including recoloring, retexturing, illumination editing, and lighting-aware compositing.

An L 1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition

“…propose that the reflectance at each pixel can be represented as a weighted average of its neighbors' reflectance values. More recently, Bonneel et al [2014] separate gradients caused by reflectance and shading with a hybrid L2 and Lp optimization.…”

“…The same set of tools are also supported in Bonneel et al 2014]. Although automatic algorithms are better suited for processing large-scale datasets, their results inevitably have inaccuracies.…”

A Comprehensive Multi-Illuminant Dataset for Benchmarking of the Intrinsic Image Algorithms

“…Intrinsic video decomposition methods are becoming a recent trend in the community [30,8]. It would be interesting to extend our dataset to dynamic scenes.…”

Intrinsic Decomposition of Image Sequences from Local Temporal Variations

“…Barron et al [1], Laffont et al [19,20], and Hauagge et al [13] leverage scene geometry, which is either captured with a depth sensor or reconstructed from multiple viewpoints. Bonneel et al [6] decompose video sequences and let users interactively refine the results, while Ye et al [35] propagate an existing decomposition to every frame of a video. Other methods decompose timelapses, i.e., sequences captured from a fixed viewpoint under varying lighting conditions.…”