Modular primitives for high-performance differentiable rendering

Laine, Samuli; Hellsten, Janne; Karras, Tero; Seol, Yeongho; Lehtinen, Jaakko; Aila, Timo

doi:10.1145/3414685.3417861

Cited by 216 publications

(131 citation statements)

References 29 publications

Supporting

Mentioning

114

Contrasting

Order By: Relevance

“…As demonstrated in Table 1, our renderer is faster than SoftRas [LLCL19], PyTorch3D [RRN*20], and Mitsuba 2 [NDVZJ19] without the need to introduce bias to the gradient estimates. Nvdiffrast [LHK*20] is faster than our system but produces approximated gradients. Lastly, compared to Redner [LADL18], another differentiable renderer that produces unbiased gradients, our renderer offers better performance.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering

Luan

Zhao

Bala

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

Reconstructing the shape and appearance of real‐world objects using measured 2D images has been a long‐standing inverse rendering problem. In this paper, we introduce a new analysis‐by‐synthesis technique capable of producing high‐quality reconstructions through robust coarse‐to‐fine optimization and physics‐based differentiable rendering. Unlike most previous methods that handle geometry and reflectance largely separately, our method unifies the optimization of both by leveraging image gradients with respect to both object reflectance and geometry. To obtain physically accurate gradient estimates, we develop a new GPU‐based Monte Carlo differentiable renderer leveraging recent advances in differentiable rendering theory to offer unbiased gradients while enjoying better performance than existing tools like PyTorch3D [RRN*20] and redner [LADL18]. To further improve robustness, we utilize several shape and material priors as well as a coarse‐to‐fine optimization strategy to reconstruct geometry. Using both synthetic and real input images, we demonstrate that our technique can produce reconstructions with higher quality than previous methods.

show abstract

Section: Resultsmentioning

confidence: 99%

“… Comparison with SoftRas [LLCL19], PyTorch3D [RRN*20], Mitsuba 2 [NDVZJ19] and Nvdiffrast [LHK*20]. We render all reconstructed geometries using Phong shading and visualize depth errors (wrt.…”

Section: Resultsmentioning

confidence: 99%

Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering

Luan

Zhao

Bala

et al. 2021

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…given noisy, rendered observations of a small patch (6 × 6) of pixels. Excellent options are available [LHK*20, BLD20, NDVZJ19, ZWZ*19] for inferring point estimates of 9 given some observed (target) radiance distribution ℓ . Typically these methods use a differentiable rendering pipeline enabling the optimization of 9 via iterative back‐propagation of gradients with respect to 9.…”

Section: Results Ii: Sample Applicationsmentioning

confidence: 99%

Q‐NET: A Network for Low‐dimensional Integrals of Neural Proxies

Subr

2021

Computer Graphics Forum

View full text Add to dashboard Cite

Integrals of multidimensional functions are often estimated by averaging function values at multiple locations. The use of an approximate surrogate or proxy for the true function is useful if repeated evaluations are necessary. A proxy is even more useful if its own integral is known analytically and can be calculated practically. We design a family of fixed networks, which we call Q‐NETs, that can calculate integrals of functions represented by sigmoidal universal approximators. Q‐NETs operate on the parameters of the trained proxy and can calculate exact integrals over any subset of dimensions of the input domain. Q‐NETs also facilitate convenient recalculation of integrals without resampling the integrand or retraining the proxy, under certain transformations to the input space. We highlight the benefits of this scheme for diverse rendering applications including inverse rendering, sampled procedural noise and visualization. Q‐NETs are appealing in the following contexts: the dimensionality is low (< 10D); integrals of a sampled function need to be recalculated over different sub‐domains; the estimation of integrals needs to be decoupled from the sampling strategy such as when sparse, adaptive sampling is used; marginal functions need to be known in functional form; or when powerful Single Instruction Multiple Data/Thread (SIMD/SIMT) pipelines are available.

show abstract

“…Recently, there are strong interests in graphics, vision, and machine learning communities to build fully differentiable renderers. Some methods ignore geometry derivatives [Gkioulekas et al 2013;Nimier-David et al 2019], some approximate the Dirac delta contributions [de La Gorce et al 2011;Kato et al 2018;Loper and Black 2014], some smooth out the discontinuities [Liu et al 2019], and some apply smooth postprocessing using the geometry buffer [Laine et al 2020]. Meanwhile, other methods derive the correct derivatives using Dirac deltas [Li et al 2018a], reparametrization [Loubet et al 2019], or Reynolds transport theorem [Bangaru et al 2020;Li et al 2020b;Roger et al 2005;Zhang et al 2020Zhang et al , 2019.…”

Section: Related Workmentioning

confidence: 99%

Systematically differentiating parametric discontinuities

Bangaru¹,

Michel²,

Mu³

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

Emerging research in computer graphics, inverse problems, and machine learning requires us to differentiate and optimize parametric discontinuities. These discontinuities appear in object boundaries, occlusion, contact, and sudden change over time. In many domains, such as rendering and physics simulation, we differentiate the parameters of models that are expressed as integrals over discontinuous functions. Ignoring the discontinuities during differentiation often has a significant impact on the optimization process. Previous approaches either apply specialized hand-derived solutions, smooth out the discontinuities, or rely on incorrect automatic differentiation. We propose a systematic approach to differentiating integrals with discontinuous integrands, by developing a new differentiable programming language. We introduce integration as a language primitive and account for the Dirac delta contribution from differentiating parametric discontinuities in the integrand. We formally define the language semantics and prove the correctness and closure under the differentiation, allowing the generation of gradients and higher-order derivatives. We also build a system, Teg, implementing these semantics. Our approach is widely applicable to a variety of tasks, including image stylization, fitting shader parameters, trajectory optimization, and optimizing physical designs.

show abstract

Modular primitives for high-performance differentiable rendering

Cited by 216 publications

References 29 publications

Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering

Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering

Q‐NET: A Network for Low‐dimensional Integrals of Neural Proxies

Systematically differentiating parametric discontinuities

Contact Info

Product

Resources

About