Single-image SVBRDF capture with a rendering-aware deep network

Abstract

Fig. 1. From a single flash photograph of a material sample (insets), our deep learning approach predicts a spatially-varying BRDF. See supplemental materials for animations with a moving light.Texture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional re ectance distribution functions (SVBRDFs) from a single image based on such cues has challenged researchers in computer graphics for decades. We tackle lightweight appearance capture by training a deep neural network to automatically extract and make sense of these visual cues. Once trained, our network is capable of recovering per-pixel normal, di use albedo, specular albedo and specular roughness from a single picture of a at surface lit by a hand-held ash. We achieve this goal by introducing several innovations on training data acquisition and network design. For training, we leverage a large dataset of artist-created, procedural SVBRDFs which we sample and render under multiple lighting directions. We further amplify the data by material mixing to cover a wide diversity of shading e ects, which allows our network to work across many material classes. Motivated by the observation that distant regions of a material sample o en o er complementary visual cues, we design a network that combines an encoder-decoder convolutional track for local feature extraction with a fully-connected track for global feature extraction and propagation. Many important material e ects are view-dependent, and as such ambiguous when observed in a single image. We tackle this challenge by de ning the loss as a di erentiable SVBRDF similarity metric that compares the renderings of the predicted maps against renderings of the ground truth from several lighting and viewing directions. Combined together, these novel ingredients bring clear improvement over state of the art methods for single-shot capture of spatially varying BRDFs.is is the author's version of the work. It is posted here for your personal use. Not for redistribution. e de nitive version was published in ACM Trans.

On‐Site Example‐Based Material Appearance Acquisition

“…A number of methods have been proposed that rely on deep learning to estimate the reflectance properties and meso‐structure from a single photograph under flash lighting [DAD∗18, LSC18, AAL16, LXR∗18] or under natural lighting [LDPT17, YLD∗18]. While promising, these methods either control the incident lighting, and/or are limited to planar samples only.…”

Learned Fitting of Spatially Varying BRDFs

“…Though they achieve impressive results for separating all these coupled effects, their Cook‐Torrance‐like SVBRDF lacks a specular albedo and the authors assign a fixed value for the Fresnel F 0 parameter. Deschaintre et al [DAD∗ 18] tackle the problem of limited datasets by creating new materials from permutations and combinations of procedural SVBRDF models obtained from Allegorithmic Substance share [all19]. They use their 200,000 generated SVBRDFs to train an encoder‐decoder network, with an additional global feature track to bypass and fuse features that are removed by instance normalizations in their main network.…”

“…High quality SVBRDF fitting via (a) non‐linear optimization vs. (b) our deep learning approach on calibrated measurements vs. (c) a state‐of‐the‐art single‐shot SVBRDF estimation based on a single, uncalibrated photograph [DAD∗18]. Though the single‐shot method's result obtained from a single photograph are impressive, the limitations of such approaches are obvious.…”

“…Learning based methods have been previously applied in the field of appearance capture. In the last years a series of works that utilize deep learning models to extract reflectance from varying kinds of input images has been published [AAL16; LDPT17; YLD∗18; KCW∗18; LSC18; LXR∗18; DAD∗18; VCGL19]. The trend is to feed very sparsely measured inputs into a network, usually one or a few photographs, to obtain an estimate of the observed surface reflectance.…”

“…To the best of our knowledge, this represents the state of the art for high‐quality material fits. Furthermore, we provide some comparisons against a state of the art few‐shot method [DAD∗ 18].…”