Unified Neural Encoding of BTFs

Abstract

Realistic rendering using discrete reflectance measurements is challenging, because arbitrary directions on the light and view hemispheres are queried at render time, incurring large memory requirements and the need for interpolation. This explains the desire for compact and continuously parametrized models akin to analytic BRDFs; however, fitting BRDF parameters to complex data such as BTF texels can prove challenging, as models tend to describe restricted function spaces that cannot encompass real‐world behavior. Recent advances in this area have increasingly relied on neural representations that are trained to reproduce acquired reflectance data. The associated training process is extremely costly and must typically be repeated for each material. Inspired by autoencoders, we propose a unified network architecture that is trained on a variety of materials, and which projects reflectance measurements to a shared latent parameter space. Similarly to SVBRDF fitting, real‐world materials are represented by parameter maps, and the decoder network is analog to the analytic BRDF expression (also parametrized on light and view directions for practical rendering application). With this approach, encoding and decoding materials becomes a simple matter of evaluating the network. We train and validate on BTF datasets of the University of Bonn, but there are no prerequisites on either the number of angular reflectance samples, or the sample positions. Additionally, we show that the latent space is well‐behaved and can be sampled from, for applications such as mipmapping and texture synthesis.

NeuMIP

“…6 when compared to the reference, both using a per-pixel MSE (Mean Squared Error) and perceptual LPIPS (Learned Perceptual Image Patch Similarity) scores. Our scores, shown in Table 2, are consistently better than both Rainer et al [2019] and Rainer et al [2020] .…”

“…The materials are mapped to a plane, viewed at an angle and lit by a single light slightly to the left. Our baseline method already outperforms encoders of Rainer et al [2019] and Rainer et al [2020] methods, despite being trained with fewer BTF queries. We believe this is due to our decoder-only architecture, which can adapt to the material and benefits from a stochastic distribution of the input BTF queries, and our improved training techniques (especially the progressively decaying spatial Gaussian blur).…”

“…In Figure 6, we show results rendered on a flat plane, with camera and directional light at an angle. We compare several different methods: Rainer et al [2019], Rainer et al [2020], our baseline method (without neural offset), our full method (with neural offset), and a ground truth computed by path-tracing of the synthetic material structure. The materials are mapped to a plane, viewed at an angle and lit by a single light slightly to the left.…”

“…The multi-resolution nature of our solution also helps. On the other hand, Rainer et al [2020]'s solution has the benefit of very fast encoding in case the queries are already in the Fig. 7.…”

“…Recent related works [Rainer et al 2020[Rainer et al , 2019 also use a neural network to efficiently compress BTFs, and we build upon these methods. However, they do not support prefiltering with arbitrary kernel sizes.…”