Radiative backpropagation

Abstract

Physically based differentiable rendering has recently evolved into a powerful tool for solving inverse problems involving light. Methods in this area perform a differentiable simulation of the physical process of light transport and scattering to estimate partial derivatives relating scene parameters to pixels in the rendered image. Together with gradient-based optimization, such algorithms have interesting applications in diverse disciplines, e.g., to improve the reconstruction of 3D scenes, while accounting for interreflection and transparency, or to design meta-materials with specified optical properties. The most versatile differentiable rendering algorithms rely on reverse-mode differentiation to compute all requested derivatives at once, enabling optimization of scene descriptions with millions of free parameters. However, a severe limitation of the reverse-mode approach is that it requires a detailed transcript of the computation that is subsequently replayed to back-propagate derivatives to the scene parameters. The transcript of typical renderings is extremely large, exceeding the available system memory by many orders of magnitude, hence current methods are limited to simple scenes rendered at low resolutions and sample counts. We introduce radiative backpropagation , a fundamentally different approach to differentiable rendering that does not require a transcript, greatly improving its scalability and efficiency. Our main insight is that reverse-mode propagation through a rendering algorithm can be interpreted as the solution of a continuous transport problem involving the partial derivative of radiance with respect to the optimization objective. This quantity is "emitted" by sensors, "scattered" by the scene, and eventually "received" by objects with differentiable parameters. Differentiable rendering then decomposes into two separate primal and adjoint simulation steps that scale to complex scenes rendered at high resolutions. We also investigated biased variants of this algorithm and find that they considerably improve both runtime and convergence speed. We showcase an efficient GPU implementation of radiative backpropagation and compare its performance and the quality of its gradients to prior work.

Curriculum Learning for ab initio Deep Learned Refractive Optics

“…A straightforward implementation of end-to-end training of differentiable ray-tracing using automatic differentiation consumes an infeasibly large amount of computer memory. Existing approaches either compute adjoint derivatives [19,28] in the forward pass, simplify intermediate computations [31], or use small sensor resolutions and sampling rate [26]. Neither of these approaches provides a satisfactory solution to large-scale, high resolution deep lens design of complex optical systems.…”

dO: A Differentiable Engine for Deep Lens Design of Computational Imaging Systems

“…A similar memory-efficient approach was proposed in Ref. [46]. The separability property reflects by introducing an intermediate derivative image ∆I to Eq.…”

“…Compared to model-based (or, physics-based) learning scenarios [44], our applications require the image formation to be computed in a Monte Carlo fashion and cannot be explicitly stated, and thus memoryefficient techniques rely on known image formation models like [44] are not directly applicable. Similar memory-hunger issue exists also in differentiable rendering for end-to-end learning [37], [45], and solutions have been proposed [46]. However, an optical design engine differs from a generalpurpose graphics renderer, in that optical surface geometry representations are well-parameterized surfaces (e.g., aspheric or freeform splines) rather than discrete meshes.…”

Dressi: A Hardware‐Agnostic Differentiable Renderer with Reactive Shader Packing and Soft Rasterization

“…Moreover, it achieves highly efficient computations with template meta‐programming for various data types and a retargetable just‐in‐time (JIT) compilation for AD. Radiative backpropagation [NSRJ20] is a memory‐efficient adjoint approach that reduces the computation of backward functions in continuous light transportation. Path‐space DR [ZMY ∗ 20] shows better efficiency using the new Monte Carlo estimators.…”