Physics to the Rescue: Deep Non-Line-of-Sight Reconstruction for High-Speed Imaging

Mu, Fangzhou; Mo, Sicheng; Peng, Jiayong; Liu, Xiaochun; Nam, Ji Hyun; Raghavan, Siddeshwar; Velten, Andreas; Li, Yin

doi:10.1109/tpami.2022.3203383

Cited by 15 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Error Heatmap Besides RMSE used in [MMP*22], we also test the model with error map [CWK*20] in Figure 8 and SM. With the help of the bottom color bar, we can see that the error distributions vary spatially and align with the edges and textures of the objects.…”

Section: Resultsmentioning

confidence: 99%

“…Abramson [Abr78] was the first to show a holographic capture system for transient imaging, and Kirmani [KHDR09] temporally resolved light transport measurements capturing short pulses of light before the global transport reaches steady state, which was initially proposed. Reconstruction algorithm can be classified into four main categories [MMP*22]: backprojection methods [VWG*12], wave propagation‐based methods [HXHH14], iterative optimization methods [WLH*21], and geometry‐based methods [XNK*19]. SPIRAL‐3D [WLH*21] obtains an approximate solution of the inverse problem with LASSO‐type optimization.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Multi‐scale Iterative Model‐guided Unfolding Network for NLOS Reconstruction

Su,

Hong,

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Non‐line‐of‐sight (NLOS) imaging can reconstruct hidden objects by analyzing diffuse reflection of relay surfaces, and is potentially used in autonomous driving, medical imaging and national defense. Despite the challenges of low signal‐to‐noise ratio (SNR) and ill‐conditioned problem, NLOS imaging has developed rapidly in recent years. While deep neural networks have achieved impressive success in NLOS imaging, most of them lack flexibility when dealing with multiple spatial‐temporal resolution and multi‐scene images in practical applications. To bridge the gap between learning methods and physical priors, we present a novel end‐to‐end Multi‐scale Iterative Model‐guided Unfolding (MIMU), with superior performance and strong flexibility. Furthermore, we overcome the lack of real training data with a general architecture that can be trained in simulation. Unlike existing encoder‐decoder architectures and generative adversarial networks, the proposed method allows for only one trained model adaptive for various dimensions, such as various sampling time resolution, various spatial resolution and multiple channels for colorful scenes. Simulation and real‐data experiments verify that the proposed method achieves better reconstruction results both in quality and quantity than existing methods.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Multi‐scale Iterative Model‐guided Unfolding Network for NLOS Reconstruction

Su,

Hong,

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Learning-based approaches. Other methods leverage neural networks instead, such as U-net [Grau Chopite et al 2020], convolutional neural networks , or neural radiance fields [Mu et al 2022]. These learning-based methods are learned using object databases such as ShapeNet [Chang et al 2015].…”

Section: Related Workmentioning

confidence: 99%

“…In particular, the phasor field framework [Liu et al 2019] computationally transforms the data captured on the relay surface into illumination arriving at a virtual imaging aperture. This has enabled more complex imaging models (e.g., [Dove and Shapiro 2020a,b;Guillén et al 2020;), and boosted the efficiency of NLOS imaging to interactive and real-time reconstruction rates [Liao et al 2021;Mu et al 2022;]. However, these systems require careful calibration of all their parameters, including the definition of the phasor field and the particular characteristics of lasers and sensors, which makes using them a cumbersome process.…”

Section: Related Workmentioning

confidence: 99%

Self-Calibrating, Fully Differentiable NLOS Inverse Rendering

Choi,

Kim,

Choi

et al. 2023

SIGGRAPH Asia 2023 Conference Papers

View full text Add to dashboard Cite

Existing time-resolved non-line-of-sight (NLOS) imaging methods reconstruct hidden scenes by inverting the optical paths of indirect illumination measured at visible relay surfaces. These methods are prone to reconstruction artifacts due to inversion ambiguities and capture noise, which are typically mitigated through the manual selection of filtering functions and parameters. We introduce a fully-differentiable end-to-end NLOS inverse rendering pipeline that self-calibrates the imaging parameters during the reconstruction of hidden scenes, using as input only the measured illumination while working both in the time and frequency domains. Our pipeline extracts a geometric representation of the hidden scene from NLOS volumetric intensities and estimates the time-resolved illumination at the relay wall produced by such geometric information using differentiable transient rendering. We then use gradient descent to optimize imaging parameters by minimizing the error between our simulated time-resolved illumination and the measured illumination. Our end-to-end differentiable pipeline couples diffraction-based volumetric NLOS reconstruction with path-space light transport and a simple ray marching technique to extract detailed, dense sets of surface points and normals of hidden scenes. We demonstrate the robustness of our method to consistently reconstruct geometry and albedo, even under significant noise levels. CCS CONCEPTS• Computing methodologies → Image and video acquisition.

show abstract

“…Previous NLOS methods represent the scene by either volumetric albedo [14], [15], volumetric feature [42], [43] or parametric surface [38], [44]. Recently, neural-based approaches have made remarkable progress [2], [45].…”

Section: Related Workmentioning

confidence: 99%

Onsite Non-Line-of-Sight Imaging via Online Calibration

Pan

Gao

et al. 2022

IEEE Photonics J.

View full text Add to dashboard Cite

There has been increasing interest in deploying non-line-of-sight (NLOS) imaging systems for recovering objects hidden behind corners. Existing solutions need to calibrate the imaging system using auxiliary apparatus and additional detectors. We present an online calibration technique that directly decouples the transients, which are acquired by onsite scanning on a relay surface, into line-of-sight (LOS) and hidden components. We use the former to directly (re-)calibrate the system upon changes of scene−surface configurations, scannable regions, and sampling patterns, and the latter for hidden object recovery via spatial-, frequency-, or learning-based techniques. We also calculate a Gamma map from the LOS component to preview calibration effects for accurate transient measurements.The entire process of our calibration for 64 scanning points takes no more than 14 seconds on an Intel i7-6600H CPU. In particular, our technique avoids using auxiliary calibration tools such as mirrors or checkerboards and supports both uniform and non-uniform sampling in an onsite NLOS imaging system. Comprehensive experiments via calibration evaluation and NLOS reconstruction demonstrate the efficiency and effectiveness of our solution. Besides, we have made our data and code open-source on GitHub to the research community.

show abstract

Physics to the Rescue: Deep Non-Line-of-Sight Reconstruction for High-Speed Imaging

Cited by 15 publications

References 0 publications

Multi‐scale Iterative Model‐guided Unfolding Network for NLOS Reconstruction

Multi‐scale Iterative Model‐guided Unfolding Network for NLOS Reconstruction

Self-Calibrating, Fully Differentiable NLOS Inverse Rendering

Onsite Non-Line-of-Sight Imaging via Online Calibration

Contact Info

Product

Resources

About