Differentiable signed distance function rendering

Abstract

Physically-based differentiable rendering has recently emerged as an attractive new technique for solving inverse problems that recover complete 3D scene representations from images. The inversion of shape parameters is of particular interest but also poses severe challenges: shapes are intertwined with visibility, whose discontinuous nature introduces severe bias in computed derivatives unless costly precautions are taken. Shape representations like triangle meshes suffer from additional difficulties, since the continuous optimization of mesh parameters cannot introduce topological changes. One common solution to these difficulties entails representing shapes using signed distance functions (SDFs) and gradually adapting their zero level set during optimization. Previous differentiable rendering of SDFs did not fully account for visibility gradients and required the use of mask or silhouette supervision, or discretization into a triangle mesh. In this article, we show how to extend the commonly used sphere tracing algorithm so that it additionally outputs a reparameterization that provides the means to compute accurate shape parameter derivatives. At a high level, this resembles techniques for differentiable mesh rendering, though we show that the SDF representation admits a particularly efficient reparameterization that outperforms prior work. Our experiments demonstrate the reconstruction of (synthetic) objects without complex regularization or priors, using only a per-pixel RGB loss.

Onsite Non-Line-of-Sight Imaging via Online Calibration

“…NeTF [2] applies a density field to represent the scene, which fails to acquire a high-quality surface. Concurrent to our work, NLOS-NeuS [45] represent NLOS scenes with signed distance fields (SDF) [18], [46], which shows better surface recovery. Different from them, our technique incorporates a more general level set representation, on which we develop a unified transient rendering formulation under both LOS and NLOS settings.…”

Ray‐aligned Occupancy Map Array for Fast Approximate Ray Tracing

“…The purpose of ROMA is to provide a different way to perform ray tracing. Though much faster in building and tracing, ROMA is not designed as an alternative to DFs, for that DFs offer other convenient properties other than ray tracing, e.g., the simplicity to compute surface normal, differentiability [VSJ22] and their unique way to compute soft shadows [Wri15].…”

AI-Assisted Design for Intangible Cultural Heritage: A Study on the Tujia Hand-Waving Dance

“…In terms of 2D posture reconstruction, previous research was based on OpenPose and image translation [1,2] to achieve the recognition and reconstruction of dance postures [3]. 3D reconstruction focuses on reconstructing pose using as little information as a single view [4,5,6,7,8,9]. Related work has initially achieved 3D reconstruction of the human body from single-view images or videos [10,11,12,13,14,15,16].…”