Node Graph Optimization Using Differentiable Proxies

We propose MatUp, an upsampling filter for material super‐resolution. Our method takes as input a low‐resolution SVBRDF and upscales its maps so that their rendering under various lighting conditions fits upsampled renderings inferred in the radiance domain with pre‐trained RGB upsamplers. We formulate our local filter as a compact Multilayer Perceptron (MLP), which acts on a small window of the input SVBRDF and is optimized using a data‐fitting loss defined over upsampled radiance at various locations. This optimization is entirely performed at the scale of a single, independent material. Doing so, MatUp leverages the reconstruction capabilities acquired over large collections of natural images by pre‐trained RGB models and provides regularization over self‐similar structures. In particular, our light‐weight neural filter avoids retraining complex architectures from scratch or accessing any large collection of low/high resolution material pairs – which do not actually exist at the scale RGB upsamplers are trained with. As a result, MatUp provides fine and coherent details in the upscaled material maps, as shown in the extensive evaluation we provide.

Section: Related Workmentioning

confidence: 99%

MatUp: Repurposing Image Upsamplers for SVBRDFs

Gauthier,

Kerbl,

Levallois

et al. 2024

“…[SLH*20] and Hu et al. [HGH*22] used differentiable procedural material definitions (in the form of hand‐written programs or node graphs, respectively) to optimize parameters to match a user‐provided photograph. To alleviate the need for a pre‐existing procedural material database, Hu et al.…”

Section: Related Workmentioning

confidence: 99%

A Semi‐Procedural Convolutional Material Prior

Zhou

Hašan

Deschaintre

et al. 2023

Self Cite

Lightweight material capture methods require a material prior, defining the subspace of plausible textures within the large space of unconstrained texel grids. Previous work has either used deep neural networks (trained on large synthetic material datasets) or procedural node graphs (constructed by expert artists) as such priors. In this paper, we propose a semi‐procedural differentiable material prior that represents materials as a set of (typically procedural) grayscale noises and patterns that are processed by a sequence of lightweight learnable convolutional filter operations. We demonstrate that the restricted structure of this architecture acts as an inductive bias on the space of material appearances, allowing us to optimize the weights of the convolutions per‐material, with no need for pre‐training on a large dataset. Combined with a differentiable rendering step and a perceptual loss, we enable single‐image tileable material capture comparable with state of the art. Our approach does not target the pixel‐perfect recovery of the material, but rather uses noises and patterns as input to match the target appearance. To achieve this, it does not require complex procedural graphs, and has a much lower complexity, computational cost and storage cost. We also enable control over the results, through changing the provided patterns and using guide maps to push the material properties towards a user‐driven objective.

“…Since not all operators can be made differentiable, only a subset of the program parameters can be optimized. Differentiable Proxies [HGH*22] tackles the problem of non‐differentiable operators by training small neural networks to approximate these operators. The networks act as proxies that are differentiable and have parameters that can be optimized with gradient descent.…”

Section: Application: Materials and Texturesmentioning

confidence: 99%

Neurosymbolic Models for Computer Graphics

Ritchie

Guerrero

Jones

et al. 2023

Self Cite

Procedural models (i.e. symbolic programs that output visual data) are a historically‐popular method for representing graphics content: vegetation, buildings, textures, etc. They offer many advantages: interpretable design parameters, stochastic variations, high‐quality outputs, compact representation, and more. But they also have some limitations, such as the difficulty of authoring a procedural model from scratch. More recently, AI‐based methods, and especially neural networks, have become popular for creating graphic content. These techniques allow users to directly specify desired properties of the artifact they want to create (via examples, constraints, or objectives), while a search, optimization, or learning algorithm takes care of the details. However, this ease of use comes at a cost, as it's often hard to interpret or manipulate these representations. In this state‐of‐the‐art report, we summarize research on neurosymbolic models in computer graphics: methods that combine the strengths of both AI and symbolic programs to represent, generate, and manipulate visual data. We survey recent work applying these techniques to represent 2D shapes, 3D shapes, and materials & textures. Along the way, we situate each prior work in a unified design space for neurosymbolic models, which helps reveal underexplored areas and opportunities for future research.