Deep Non-Line-of-Sight Reconstruction

Chopite, Javier Grau; Hullin, Matthias B.; Wand, Michael; Iseringhausen, Julian

doi:10.1109/cvpr42600.2020.00104

Cited by 26 publications

(38 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, benefiting from the proposed domain adversarial adaption approaches, STIN achieves excellent generalization performance in realistic scenarios. An interesting future direction would be applying STIN and transfer learning approaches to photon-efficient multi-surface imaging [37] and non-line-of-sight imaging [5] in realistic scenarios.…”

Section: Discussionmentioning

confidence: 99%

Deep Domain Adversarial Adaptation for Photon-efficient Imaging

Chen¹,

Yao²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

In single-photon LiDAR, photon-efficient imaging captures the 3D structure of a scene by only several detected signal photons per pixel. The existing deep learning models for this task are trained on simulated datasets, which poses the domain shift challenge when applied to realistic scenarios. In this paper, we propose a spatiotemporal inception network (STIN) for photon-efficient imaging, which is able to precisely predict the depth from a sparse and highnoise photon counting histogram by fully exploiting spatial and temporal information. Then the domain adversarial adaptation frameworks, including domain-adversarial neural network and adversarial discriminative domain adaptation, are effectively applied to STIN to alleviate the domain shift problem for realistic applications. Comprehensive experiments on the simulated data generated from the NYU v2 and the Middlebury datasets demonstrate that STIN outperforms the state-of-the-art models at low signal-tobackground ratios from 2:10 to 2:100. Moreover, experimental results on the real-world dataset captured by the single-photon imaging prototype show that the STIN with domain adversarial training achieves better generalization performance compared with the state-of-the-arts as well as the baseline STIN trained by simulated data.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep Domain Adversarial Adaptation for Photon-efficient Imaging

Chen¹,

Yao²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In contrast, our model is trained with only 2D supervisions and does not internally operate via a 3D feature space. This means that our model is much easier to train and requires less training samples than the existing ones [8,6]. It also enables us to create training pairs of transent NLOS data and photorealistic hidden scene images.…”

Section: Nlos Reconstruction Methodsmentioning

confidence: 99%

“…Deep learning has demonstrated recent success in computational imaging [2,7,43,42,37]. Chopite et al [8] propose to address the inverse problem with deep learning by first simulating plenty of training data according to confocal imaging system and then train a U-Net to recover the geometry of hidden objects. Chen et al [6] propose a deep framework for both reconstruction and recognition tasks from NLOS measurements, which achieves the best performance so far.…”

Section: Nlos Reconstruction Methodsmentioning

confidence: 99%

Towards Non-Line-of-Sight Photography

Peng¹,

Mu²,

Nam³

et al. 2021

Preprint

View full text Add to dashboard Cite

Non-line-of-sight (NLOS) imaging is based on capturing the multi-bounce indirect reflections from the hidden objects. Active NLOS imaging systems rely on the capture of the time of flight of light through the scene, and have shown great promise for the accurate and robust reconstruction of hidden scenes without the need for specialized scene setups and prior assumptions. Despite that existing methods can reconstruct 3D geometries of the hidden scene with excellent depth resolution, accurately recovering object textures and appearance with high lateral resolution remains an challenging problem. In this work, we propose a new problem formulation, called NLOS photography, to specifically address this deficiency. Rather than performing an intermediate estimate of the 3D scene geometry, our method follows a data-driven approach and directly reconstructs 2D images of a NLOS scene that closely resemble the pictures taken with a conventional camera from the location of the relay wall. This formulation largely simplifies the challenging reconstruction problem by bypassing the explicit modeling of 3D geometry, and enables the learning of a deep model with a relatively small training dataset. The results are NLOS reconstructions of unprecedented lateral resolution and image quality.

show abstract

“…Such ability to see occluded parts of the scene would have numerous obvious benefits in traffic safety, search and rescue, healthcare (endoscopy), and defense, but has yet to find its way into practical applications. While a large body of work has been dedicated to the challenge of reconstructing detailed scene geometry [Arellano et al 2017;Buttafava et al 2015;Grau Chopite et al 2020;Heide et al 2019;Liu et al 2019;Tsai et al 2019;Velten et al 2012], some applications do not require a full 3D reconstruction. Often, it could be sufficient to be able to detect objects and track their motion.…”

Section: Related Workmentioning

confidence: 99%

Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging

et al. 2021

Self Cite

View full text Add to dashboard Cite

Fig. 1. We propose the use of cheap, small off-the-shelf distance sensors (far left) for a variety of computational imaging and vision tasks, and demonstrate and evaluate their capabilities in emerging sensing applications like (from left to right) material classification, non-line-of-sight tracking, and depth imaging.Time-correlated imaging is an emerging sensing modality that has been shown to enable promising application scenarios, including lidar ranging, fluorescence lifetime imaging, and even non-line-of-sight sensing. A leading technology for obtaining time-correlated light measurements are singlephoton avalanche diodes (SPADs), which are extremely sensitive and capable of temporal resolution on the order of tens of picoseconds. However, the rare and expensive optical setups used by researchers have so far prohibited these novel sensing techniques from entering the mass market. Fortunately, SPADs also exist in a radically cheaper and more power-efficient version that has been widely deployed as proximity sensors in mobile devices for almost a decade. These commodity SPAD sensors can be obtained at a mere few cents per detector pixel. However, their inferior data quality and severe technical drawbacks compared to their high-end counterparts necessitate the use of additional optics and suitable processing algorithms. In this paper, we adopt an existing evaluation platform for commodity SPAD sensors, and modify it to unlock time-of-flight (ToF) histogramming and hence computational imaging. Based on this platform, we develop and demonstrate a family of hardware/software systems that, for the first time, implement applications that had so far been limited to significantly more advanced, higher-priced setups: direct ToF depth imaging, non-line-of-sight object tracking, and material classification.

show abstract

Deep Non-Line-of-Sight Reconstruction

Cited by 26 publications

References 24 publications

Deep Domain Adversarial Adaptation for Photon-efficient Imaging

Deep Domain Adversarial Adaptation for Photon-efficient Imaging

Towards Non-Line-of-Sight Photography

Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging

Contact Info

Product

Resources

About