Controlling Material Appearance by Examples

Hu, Yiwei; Hašan, Miloš; Guerrero, Paul; Rushmeier, Holly; Deschaintre, Valentin

doi:10.1111/cgf.14591

Cited by 13 publications

(6 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Material acquisition has been an active research area for decades [GGG*16], and recent work has been focusing on lightweight acquisition, typically using deep learning for re‐construction. Different approaches were proposed to capture material from a single [DAD*18, GLT*21, LSC18, ZK21, MRR*22, VPS21], multiple [DAD*19, GLD*19, GSH*20, HHG*22] photographs or a video [YDPG21]. These methods focus on the recovery of parameters maps per pixel and do not allow for much post‐acquisition control.…”

Section: Related Workmentioning

confidence: 99%

A Semi‐Procedural Convolutional Material Prior

Zhou

Hašan

Deschaintre

et al. 2023

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

Lightweight material capture methods require a material prior, defining the subspace of plausible textures within the large space of unconstrained texel grids. Previous work has either used deep neural networks (trained on large synthetic material datasets) or procedural node graphs (constructed by expert artists) as such priors. In this paper, we propose a semi‐procedural differentiable material prior that represents materials as a set of (typically procedural) grayscale noises and patterns that are processed by a sequence of lightweight learnable convolutional filter operations. We demonstrate that the restricted structure of this architecture acts as an inductive bias on the space of material appearances, allowing us to optimize the weights of the convolutions per‐material, with no need for pre‐training on a large dataset. Combined with a differentiable rendering step and a perceptual loss, we enable single‐image tileable material capture comparable with state of the art. Our approach does not target the pixel‐perfect recovery of the material, but rather uses noises and patterns as input to match the target appearance. To achieve this, it does not require complex procedural graphs, and has a much lower complexity, computational cost and storage cost. We also enable control over the results, through changing the provided patterns and using guide maps to push the material properties towards a user‐driven objective.

show abstract

Section: Related Workmentioning

confidence: 99%

A Semi‐Procedural Convolutional Material Prior

Zhou

Hašan

Deschaintre

et al. 2023

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

show abstract

“…There has also been multiple concurrent works that generate materials conditioned on text prompts. [Hu et al(2023)] utilise an extensive material dataset to generate high quality material node graphs using an autoregressive model conditioned on a CLIP text or image embedding. [Deschaintre et al(2023)] use a large dataset of fabric materials and descriptions to fine‐tune CLIP for material retrieval.…”

Section: Related Workmentioning

confidence: 99%

Generating Parametric BRDFs from Natural Language Descriptions

Memery,

Cedron,

Subr

2023

Computer Graphics Forum

View full text Add to dashboard Cite

Artistic authoring of 3D environments is a laborious enterprise that also requires skilled content creators. There have been impressive improvements in using machine learning to address different aspects of generating 3D content, such as generating meshes, arranging geometry, synthesizing textures, etc. In this paper we develop a model to generate Bidirectional Reflectance Distribution Functions (BRDFs) from descriptive textual prompts. BRDFs are four dimensional probability distributions that characterize the interaction of light with surface materials. They are either represented parametrically, or by tabulating the probability density associated with every pair of incident and outgoing angles. The former lends itself to artistic editing while the latter is used when measuring the appearance of real materials. Numerous works have focused on hypothesizing BRDF models from images of materials. We learn a mapping from textual descriptions of materials to parametric BRDFs. Our model is first trained using a semi‐supervised approach before being tuned via an unsupervised scheme. Although our model is general, in this paper we specifically generate parameters for MDL materials, conditioned on natural language descriptions, within NVIDIA's Omniverse platform. This enables use cases such as real‐time text prompts to change materials of objects in 3D environments such as “dull plastic” or “shiny iron”. Since the output of our model is a parametric BRDF, rather than an image of the material, it may be used to render materials using any shape under arbitrarily specified viewing and lighting conditions.

show abstract

“…Henzler et al [2021] employ a convolutional neural network, conditioned on a latent code from a learned space, to convert a random noise field into a random non-repeating field of BRDFs that match the appearance of a flash-lit photograph of a stationary material. Inspired by MaterialGAN [Guo et al 2020b], and Hu et al [2022a] introduce tileable material GANs that allow for spatial control through an additional guidance image. While these networks can produce some stochastic variations around the expected value, they do not effectively sample the distribution conditioned on the input image.…”

Section: Related Workmentioning

confidence: 99%