Learning Physics-Guided Face Relighting Under Directional Light

Nestmeyer, Thomas; Lalonde, Jean‐François; Matthews, Iain; Lehrmann, Andreas M.

doi:10.1109/cvpr42600.2020.00517

Cited by 94 publications

(91 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, unlike previous works [Sun et al 2019;Zhou et al 2019] that train relighting networks in an end-to-end fashion, we found that by explicitly modeling multiple reflectance channels of facial albedo, geometry and lighting effects, we can actually better generate some challenging effects, i.e., specular and shadow, given high quality facial geometry and an advanced lighting model. Recently, Nestmeyer et al [2020] also explicitly model the shadow and specular and achieve similar conclusions. However, their method mainly focuses on image relighting under directional lightings.…”

Section: Introductionsupporting

confidence: 53%

“…We also render the albedo image under the uniform lighting environment. Although multiplicative shadow maps are often used in previous relighting methods [Nestmeyer et al 2020], we opt to employ additive shadow maps as additive shadow will ease gradient backpropagation during network optimization. Because our proposed method focuses on relighting portraits, we set the background pixels to 0 during rendering.…”

Section: Multiple Reflectance Channel Renderingmentioning

confidence: 99%

“…lighting normalization [Nagano et al 2019;Zhang et al 2020b], shadow removal [Zhang et al 2020a]. Very recently, several methods are proposed to speed up the numerical optimization [Nestmeyer et al 2020;Sengupta et al 2018;Shu et al 2017b;Sun et al 2019;Zhou et al 2019]. The methods [Sengupta et al 2018;Shu et al 2017b] relight an facial image by performing an image intrinsic decomposition via deep neural networks.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Single image portrait relighting via explicit multiple reflectance channel modeling

Wang

Ming³

et al. 2020

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

Section: Introductionsupporting

confidence: 53%

Section: Multiple Reflectance Channel Renderingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Single image portrait relighting via explicit multiple reflectance channel modeling

Wang

Ming³

et al. 2020

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

“…The network consists of two independent encoders that extract features from a multi-illuminant image A and a light direction matrix M i separately and a common decoder to output an image D i that is lit by one light and whose direction is represented in M i . Following the recent successful relighting models [8,21,22], a variation of U-Net [27] is used to connect two encoder blocks and a decoder block at the same hierarchical level. The size of M i is the same as that of A, and it is input to the network through Gaussian blur to facilitate feature extraction.…”

Section: Cnn-based Candidate Generationmentioning

confidence: 99%

Single-Shot Multi-light-Direction Searching on Discretized Lighting Space

2021

View full text Add to dashboard Cite

Existing methods for light direction estimation solve the problem only by presupposing certain assumptions, e.g., that only one light illuminates the object, the object has a uniform texture or smoothly varying surface, or there is a known object in the scene. However, these methods cannot be used reliably in real-world settings. We propose a framework that can be used in natural settings to estimate multi-light directions from a single image. To tackle this novel and challenging problem, we divide the lighting space into ranges and formulate the estimation problem as a problem of searching the ranges where the light is cast from a multi-illuminant image. We propose a two-step approach. First, we generate single-illuminant images, one of which assumes that there is only light on one range. Next, we select some images from the candidates that could each be a component of a multi-illuminant image under physical constraints. The experimental results demonstrated that the proposed method outperformed the direct-estimation method when the lighting space is finely divided.

show abstract

“…In other words, changing the illumination by controlling the light source is still in its fancy stage. Literature in relighting field mainly focuses on specific applications, like portrait relighting [27,32,41]. These methods require prior information (like face landmarks, geometric priors) that cannot be implemented in general scenes.…”

Section: Introductionmentioning

confidence: 99%

Deep Relighting Networks for Image Light Source Manipulation

Wang

Siu

Liu

et al. 2020

Computer Vision – ECCV 2020 Workshops

View full text Add to dashboard Cite

Manipulating the light source of given images is an interesting task and useful in various applications, including photography and cinematography. Existing methods usually require additional information like the geometric structure of the scene, which may not be available for most images. In this paper, we formulate the single image relighting task and propose a novel Deep Relighting Network (DRN) with three parts: 1) scene reconversion, which aims to reveal the primary scene structure through a deep auto-encoder network, 2) shadow prior estimation, to predict light effect from the new light direction through adversarial learning, and 3) re-renderer, to combine the primary structure with the reconstructed shadow view to form the required estimation under the target light source. Experiments show that the proposed method outperforms other possible methods, both qualitatively and quantitatively. Specifically, the proposed DRN has achieved the best PSNR in the "AIM2020 -Any to one relighting challenge" of the 2020 ECCV conference.

show abstract

Learning Physics-Guided Face Relighting Under Directional Light

Cited by 94 publications

References 45 publications

Single image portrait relighting via explicit multiple reflectance channel modeling

Single image portrait relighting via explicit multiple reflectance channel modeling

Single-Shot Multi-light-Direction Searching on Discretized Lighting Space

Deep Relighting Networks for Image Light Source Manipulation

Contact Info

Product

Resources

About