Live intrinsic video

Abstract

Intrinsic video decomposition refers to the fundamentally ambiguous task of separating a video stream into its constituent layers, in particular reflectance and shading layers. Such a decomposition is the basis for a variety of video manipulation applications, such as realistic recoloring or retexturing of objects. We present a novel variational approach to tackle this underconstrained inverse problem at real-time frame rates, which enables on-line processing of live video footage. The problem of finding the intrinsic decomposition is formulated as a mixed variational ℓ 2 - ℓ p -optimization problem based on an objective function that is specifically tailored for fast optimization. To this end, we propose a novel combination of sophisticated local spatial and global spatio-temporal priors resulting in temporally coherent decompositions at real-time frame rates without the need for explicit correspondence search. We tackle the resulting high-dimensional, non-convex optimization problem via a novel data-parallel iteratively reweighted least squares solver that runs on commodity graphics hardware. Real-time performance is obtained by combining a local-global solution strategy with hierarchical coarse-to-fine optimization. Compelling real-time augmented reality applications, such as recoloring, material editing and retexturing, are demonstrated in a live setup. Our qualitative and quantitative evaluation shows that we obtain high-quality real-time decompositions even for challenging sequences. Our method is able to outperform state-of-the-art approaches in terms of runtime and result quality -- even without user guidance such as scribbles.

Intrinsic Image Decomposition via Structure-Preserving Image Smoothing and Material Recognition

“…Therefore, this approach struggles if the assumptions of white light and Lambertian surfaces are violated. However, using RGB information, makes the method more resilient against the violation of these assumptions [9]. …”

“…They also used user-assistance for refining the results of the initial decomposition. More recently, Meka et al [9] proposed a variational approach by solving an l 2 -l p optimization problem to find the intrinsic image components. Their optimization problem includes local sparsity prior on reflectance, spatio-temporal reflectance consistency prior, reflectance clustering prior, and a data fitting term.…”

State of the Art on 3D Reconstruction with RGB‐D Cameras

“…An elegant way to directly incorporate them in the energy formulation is to solve the problem in log‐space which has the additional advantage of transforming the component‐wise product into a less involving sum and allowing for more efficient optimization techniques. This has been applied for either the scalar [MZRT16, MFZ*17] and the vector‐valued versions of the least‐squares error terms [SYJL11, LZT*12, ZTD*12, BM13]. Recently, Bonneel et al [BST*14] propose to optimize for the gradient of the log‐reflectance and shading which is another convenient reformulation of the original problem.…”

“…Recently, Bonneel et al [BST*14] propose to optimize for the gradient of the log‐reflectance and shading which is another convenient reformulation of the original problem. In addition, weighting the contribution of each sample adaptively based on the brightness of the observed color values further improves the robustness and reduces the influence of dark regions with less reliable color values [KGB14,MZRT16,MFZ*17,CK13].…”

“…Thus, each term is weighted based on a thresholded gradient image [LM71, LZT*12, ZTD*12] to allow sharp edges in the reflectance image. Instead of considering the gradient magnitude, chromaticity‐based weights have been used to robustly detect reflectance edges [SYH13, ZDI*15, MZRT16, MFZ*17]. Shen et al [SYLJ13, SYJL11] extend the original Retinex weights to also consider the observed intensity values and relax the constraint at dark regions where reflectance can be less reliably estimated and differences might not be detected accurately.…”