Temporal frequency probing for 5D transient analysis of global light transport

Abstract

a) scene (b) transient frame (t = 1.9 ns) (c) transient frame (t = 3.2 ns) (d) mesh Figure 1: (a) We use a photonic mixer device (PMD) coupled with a 2D projector to acquire "light-in-flight" images and depth maps for scenes with complex indirect transport. (b)-(c) The sharp, evolving wavefronts of light travelling along direct and caustic light paths can be seen clearly. (d) We also recover time-of-flight depth maps in a way that is not affected by strong indirect light. This indirect light mainly consists of diffuse inter-reflections for the scene in (a) but we also show experimental results where caustic light paths dominate. AbstractWe analyze light propagation in an unknown scene using projectors and cameras that operate at transient timescales. In this new photography regime, the projector emits a spatio-temporal 3D signal and the camera receives a transformed version of it, determined by the set of all light transport paths through the scene and the time delays they induce. The underlying 3D-to-3D transformation encodes scene geometry and global transport in great detail, but individual transport components (e.g., direct reflections, inter-reflections, caustics, etc.) are coupled nontrivially in both space and time.To overcome this complexity, we observe that transient light transport is always separable in the temporal frequency domain. This makes it possible to analyze transient transport one temporal frequency at a time by trivially adapting techniques from conventional projector-to-camera transport. We use this idea in a prototype that offers three never-seen-before abilities: (1) acquiring time-of-flight depth images that are robust to general indirect transport, such as interreflections and caustics; (2) distinguishing between direct views of objects and their mirror reflection; and (3) using a photonic mixer device to capture sharp, evolving wavefronts of "light-in-flight".

Imaging in scattering media using correlation image sensors and sparse convolutional coding

“…For regular scenes the (intensity modulated) radiosity leaving a single scene point integrates over a continuum of scene points, and thus in general cannot be sparse. For example, any concave object can be expected to deliver a nonsparse response, as shown by O'Toole et al [16]. Especially for imaging in scattering media, the assumption of temporal sparsity breaks down.…”

What Are Optimal Coding Functions for Time-of-Flight Imaging?

“…We assume that there is no indirect or multi-bounce light component [Freedman et al 2014;Heide et al 2013;Kadambi et al 2013;O'Toole et al 2014], i.e., light from the source bounces only once at a scene point and returns to the sensor. Then, the radiance L(p, t) incident at pixel p at time t is given as:…”

“…Multi-path propagation is an important problem for almost all active imaging systems, including continuous-wave ToF. Several methods have been proposed to mitigate the (often large) depth errors due to multi-path propagation, including epipolar imaging [O'Toole et al 2015] and high-frequency coding [Gupta et al 2015;O'Toole et al 2014]. Since dealing with multi-path is an orthogonal problem to increasing depth resolution, these methods for dealing with multi-path can be used in a complementary manner to the proposed coding schemes.…”

Selectively De-animating and Stabilizing Videos

“…As a consequence, researchers demonstrate that they can image around occluders by considering the inter-reflections of light [111]. Subsequently, researchers have developed methods that uses inexpensive time-of-flight sensors or projector and camera systems to capture light moving through space as well as imaging around occluders [55,83,54].…”