Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images

Abstract

Fig. 1. Visualizations under natural lighting of four captured 1k resolution SVBRDFs estimated using our deep inverse rendering framework. The leather material (left) is reconstructed from just 2 input photographs captured with a mobile phone camera and flash, while the other materials are recovered from 20 input photographs. In this paper we present a unified deep inverse rendering framework for estimating the spatially-varying appearance properties of a planar exemplar from an arbitrary number of input photographs, ranging from just a single photograph to many photographs. The precision of the estimated appearance scales from plausible when the input photographs fails to capture all the reflectance information, to accurate for large input sets. A key distinguishing feature of our framework is that it directly optimizes for the appearance parameters in a latent embedded space of spatially-varying appearance, such that no handcrafted heuristics are needed to regularize the optimization. This latent embedding is learned through a fully convolutional auto-encoder that has been designed to regularize the optimization. Our framework not only supports an arbitrary number of input photographs, but also at high

MaterialGAN

“…We demonstrate that our GAN-based optimization framework produces high-quality SVBRDF reconstructions from a small number (3-7) images captured under flash illumination using hand-held mobile phones, and improves upon previous state-of-the-art methods [Deschaintre et al 2019;Gao et al 2019]. In particular, it produces cleaner, more realistic material maps that better reproduce the appearance of the captured material under both input and novel lighting.…”

“…As a result, the reconstructed material parameters may not accurately reproduce the measured appearance. In contrast, Gao et al [2019] propose using an optimization-based approach in conjunction with a learned material prior. Specifically, they train a fully-convolutional auto-encoder on a large material dataset and optimize in the latent space of this auto-encoder.…”

“…We believe this is not due to any technical flaw; the method may be reaching the limit of what is possible using current feed-forward convolutional architectures and currently available datasets. Gao et al [2019] introduced an inverse rendering-based material capture approach that optimizes for material maps to minimize error with respect to the captured images. Since this is an underconstrained problem, they propose optimizing over the latent space of a learned material auto-encoder network to minimize rendering error.…”

“…MaterialGAN was trained with the dataset provided by Deschaintre et al [2018] (and also used in Gao et al [2019]). They generated this dataset by sampling the parameters of procedural material graphs from Allegorithmic Substance Share to create an initial set of 155 high-quality SVBRDFs at resolution 4096 × 4096.…”

“…Here we compare our method and Gao et al [2019]. For more results and comparisons, including with Deschaintre et al [2019], and including with and without initialization for ours and Gao's method, please refer to supplemental materials.…”