Non-line-of-sight Reconstruction Using Efficient Transient Rendering

Abstract

Object Object(a) (b) (c) Fig. 1. (a) The challenge of looking around the corner deals with the recovery of information about objects beyond the direct line of sight. In this illustration of a setting proposed by Velten et al. [2012], an unknown object is located in front of a wall, but additional obstacles occlude the object from any optical devices like light sources or cameras. Our only source of information are therefore indirect reflections off other surfaces (here, a planar "wall"). A point on the wall that is illuminated by an ultrashort laser pulse turns into an omnidirectional source of indirect light ("laser spot"). After scattering off the unknown object, some of that light arrives back at the wall, where it forms an optical "echo" or space-time response (shown are 2D slices) that can be picked up by a suitable camera. Locations on the wall can be interpreted as omnidirectional detector pixels that receive different mixtures of backscattered light contributions at different times. We assume that neither camera nor laser can directly illuminate or observe the object, leaving us with the indirect optical space-time response as the only source of information. Note that for the sake of clarity, laser source, camera, and occluder are not shown here. The complete setup is illustrated in Figure 2. (b) We propose a novel transient renderer to simulate such indirectly scattered light transport efficiently enough for use as a forward model in inverse problems. In this artistic visualization, light contributions removed by the shadow test are marked in red, and the net intensity in blue. Together with an optimization algorithm, the renderer can be used to reconstruct the geometry of objects outside the line of sight. (c) Left to right: ground-truth object geometry; reconstruction using a state-of-the-art method (ellipsoidal backprojection); reconstruction using the technique presented in this paper. Top row: BunnyGI dataset; bottom row: Mannequin1Laser dataset. Our method relies on highly efficient and near-physical forward simulation, and it exemplifies the use of computer graphics as a technical tool to solve inverse problems in other fields.Being able to see beyond the direct line of sight is an intriguing prospective and could benefit a wide variety of important applications. Recent work has demonstrated that time-resolved measurements of indirect diffuse light contain valuable information for reconstructing shape and reflectance properties of objects located around a corner. In this paper, we introduce a novel reconstruction scheme that, by design, produces solutions that are consistent with state-of-the-art physically-based rendering. Our method combines an efficient forward model (a custom renderer for time-resolved three-bounce indirect light transport) with an optimization framework to reconstruct object geometry in an analysis-by-synthesis sense. We evaluate our algorithm on a variety of synthetic and experimental input data, and show that it gracefully handles uncooperative scenes with high levels of no...

Spatial images from temporal data

“…Finally, an interesting extension would be to NLOS imaging, especially given the latest developments exploiting computational techniques for image information retrieval from temporal data [16,[40][41][42][43] and the availability of public data-sets [44,45].…”

Towards Non-Line-of-Sight Photography

“…Very little work is proposed to provide a dataset for NLOS imaging. Jarabo et al [14] introduce a framework for transient rendering, which has been adopted for NLOS data simulation in many literatures [22,12,19]. For example, Klein et al [17] introduce a synthetic dataset for various NLOS reconstruction problems yet with only 16 data samples.…”

Virtual Light Transport Matrices for Non-Line-Of-Sight Imaging

“…This line of work follows the approach proposed by Kirmani et al [16] and demonstrated experimentally by Velten and colleagues [39]: very short laser pulses illuminate a visible surface facing the hidden scene, then the scattered light reflected back onto such visible surface is captured. The hidden scene is then reconstructed using backprojection algorithms and heuristic filters [39,17,2,4], or inverting simplified light transport models [31,1,33,45,10,47,13]. These methods have usually been demonstrated in simple isolated scenes with little indirect illumination from interreflections.…”