Fast Differentiable Transient Rendering for Non-Line-of-Sight Reconstruction

Plack, Markus; Callenberg, Clara; Schneider, Monika; Hullin, Matthias B.

doi:10.1109/wacv56688.2023.00308

Cited by 4 publications

(7 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our improvements are especially noticeable in self-occluded regions and in the reproduction of detailed features. While Plack et al [2023] yields a lower RMSE in the Indonesian scene, note that it fails to reproduce large regions on the sides of the geometry. Thus, RMSE is only computed on the reconstructed regions and may not fully represent the overall accuracy of the reconstruction.…”

Section: Geometry Accuracymentioning

confidence: 94%

“…Efficiently inverting the resulting discrete light transport matrix is not trivial; many dimensionality reduction methods have been proposed [Heide et al 2019;Xin et al 2019;Young et al 2020], but they are often limited in spatial resolution (as low as 64×64 in some cases) due to memory constraints. Surface methods, in contrast, rely on inverting third-bounce light transport onto explicit representations of the geometry Plack et al 2023;Tsai et al 2019], usually starting with simple blob shapes, progressively optimizing the geometry until loss converges. In contrast, we estimate implicit geometric representations of the hidden scene based on surface points and normals by ray marching the volumetric output of NLOS imaging, inspired by recent work on neural rendering [Barron et al 2021;Mildenhall et al 2020;Niemeyer et al 2022].…”

Section: Related Workmentioning

confidence: 99%

“…In Figure 10, we compare the reconstructed geometry with surface normals in two real scenes (34 and SU) using D-LCT [Young et al 2020], NeTF [Shen et al 2021], a differentiable rendering approach [Plack et al 2023], and our method. Existing methods fail to reproduce detailed surface features in both scenes, such as the subtle changes in depth of the numbers.…”

Section: Geometry Accuracymentioning

confidence: 99%

“…In Figure 11, we provide quantitative comparisons between our estimated geometry and the geometry obtained from D-LCT [Young et al 2020], NeTF [Shen et al 2021] and Plack et al [2023] for three synthetic scenes, Dragon, Erato, and Indonesian, using the Hausdorff distance map as an objective metric. In terms of geometric accuracy, we outperform all three methods in Erato, and Dragon, as shown in the RMSE table.…”

Section: Geometry Accuracymentioning

confidence: 99%

See 3 more Smart Citations

Self-Calibrating, Fully Differentiable NLOS Inverse Rendering

Choi,

Kim,

Choi

et al. 2023

SIGGRAPH Asia 2023 Conference Papers

View full text Add to dashboard Cite

Existing time-resolved non-line-of-sight (NLOS) imaging methods reconstruct hidden scenes by inverting the optical paths of indirect illumination measured at visible relay surfaces. These methods are prone to reconstruction artifacts due to inversion ambiguities and capture noise, which are typically mitigated through the manual selection of filtering functions and parameters. We introduce a fully-differentiable end-to-end NLOS inverse rendering pipeline that self-calibrates the imaging parameters during the reconstruction of hidden scenes, using as input only the measured illumination while working both in the time and frequency domains. Our pipeline extracts a geometric representation of the hidden scene from NLOS volumetric intensities and estimates the time-resolved illumination at the relay wall produced by such geometric information using differentiable transient rendering. We then use gradient descent to optimize imaging parameters by minimizing the error between our simulated time-resolved illumination and the measured illumination. Our end-to-end differentiable pipeline couples diffraction-based volumetric NLOS reconstruction with path-space light transport and a simple ray marching technique to extract detailed, dense sets of surface points and normals of hidden scenes. We demonstrate the robustness of our method to consistently reconstruct geometry and albedo, even under significant noise levels. CCS CONCEPTS• Computing methodologies → Image and video acquisition.

show abstract

Section: Geometry Accuracymentioning

confidence: 94%

Section: Related Workmentioning

confidence: 99%

Section: Geometry Accuracymentioning

confidence: 99%

Section: Geometry Accuracymentioning

confidence: 99%

See 2 more Smart Citations

Self-Calibrating, Fully Differentiable NLOS Inverse Rendering

Choi,

Kim,

Choi

et al. 2023

SIGGRAPH Asia 2023 Conference Papers

View full text Add to dashboard Cite

show abstract

“…In particular, transient imaging is useful for nonline-of-sight (NLOS) reconstruction [Ahn et al 2019;Arellano et al 2017;Lindell et al 2019b;Liu et al 2019;Pediredla et al 2019;Velten et al 2012]. Analysis-by-synthesis optimization has been effective for NLOS problems including differentiable transient rendering [Iseringhausen and Hullin 2020;Plack et al 2023;Wu et al 2021;Yi et al 2021] and even utilized for conventional cameras [Chandran and Jayasuriya 2019;Chen et al 2019]. While there are interesting connections between transient/NLOS imaging and SAS, more research is needed to connect these domains.…”

Section: Transient and Non-line-of-sight Imagingmentioning

confidence: 99%

Neural Volumetric Reconstruction for Coherent Synthetic Aperture Sonar

et al. 2023

View full text Add to dashboard Cite

Synthetic aperture sonar (SAS) measures a scene from multiple views in order to increase the resolution of reconstructed imagery. Image reconstruction methods for SAS coherently combine measurements to focus acoustic energy onto the scene. However, image formation is typically under-constrained due to a limited number of measurements and bandlimited hardware, which limits the capabilities of existing reconstruction methods. To help meet these challenges, we design an analysis-by-synthesis optimization that leverages recent advances in neural rendering to perform coherent SAS imaging. Our optimization enables us to incorporate physics-based constraints and scene priors into the image formation process. We validate our method on simulation and experimental results captured in both air and water. We demonstrate both quantitatively and qualitatively that our method typically produces superior reconstructions than existing approaches. We share code and data for reproducibility.

show abstract