Generative modelling of BRDF textures from flash images

Abstract

We learn a latent space for easy capture, consistent interpolation, and efficient reproduction of visual material appearance. When users provide a photo of a stationary natural material captured under flashlight illumination, first it is converted into a latent material code. Then, in the second step, conditioned on the material code, our method produces an infinite and diverse spatial field of BRDF model parameters (diffuse albedo, normals, roughness, specular albedo) that subsequently allows rendering in complex scenes and illuminations, matching the appearance of the input photograph. Technically, we jointly embed all flash images into a latent space using a convolutional encoder, and -conditioned on these latent codes- convert random spatial fields into fields of BRDF parameters using a convolutional neural network (CNN). We condition these BRDF parameters to match the visual characteristics (statistics and spectra of visual features) of the input under matching light. A user study compares our approach favorably to previous work, even those with access to BRDF supervision. Project webpage: https://henzler.github.io/publication/neuralmaterial/.

Controlling Material Appearance by Examples

“…Aittala et al [AAL16] proposed using a single flash image of a stationary material to reconstruct a patch of it through neural guided optimization. Recently, deep learning was used to improve single [LDPT17,DAD∗18,HDMR21, GLT∗21, ZK21] and few‐images [DAD∗19, GSH∗20, GLD∗19, YDPG21] material acquisition. These methods recover 2D material maps based on an analytical BRDF model e.g.…”

Node Graph Optimization Using Differentiable Proxies

“…Traditionally, dozens to thousands of images were required to sample the light-view space as described in the extensive survey by Guarnera et al [2016]. More recently, deep neural networks were used to improve reconstruction from a single image [Deschaintre et al 2018;Henzler et al 2021;Zhou and Kalantari 2021] and from a small number of images [Deschaintre et al 2019[Deschaintre et al , 2020Gao et al 2019;Guo et al 2020b;Ye et al 2021]. These methods can be separated into two categories.…”

“…These methods can be separated into two categories. The first category relies on a single forward inference to recover the material parameters using an encoder/decoder architecture [Deschaintre et al 2018[Deschaintre et al , 2019Ye et al 2021;Zhou and Kalantari 2021], while the second optimizes the latent space of a pre-trained decoder network [Gao et al 2019;Guo et al 2020b;Henzler et al 2021]. While these methods can recover material parameters, they primarily focus on the reconstruction of material parameter pixel maps with their limited resolution and editability.…”

SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections

“…Casual estimation enables on-site material acquisition with simple cameras and a co-located camera flash. These techniques often constrain the problem to planar surfaces with either a single shot [1,8,17,23,32,47], few-shot [1] or multi-shot [2,9,[18][19][20] captures. This casual capture setup can also be extended to a joint BRDF and shape reconstruction [5-7, 10, 25, 43, 47, 61] or entire scenes [33,51].…”