Depth Selective Camera: A Direct, On-Chip, Programmable Technique for Depth Selectivity in Photography

Abstract

Time of flight (ToF) cameras use a temporally modulated light source and measure correlation between the reflected light and a sensor modulation pattern, in order to infer scene depth. In this paper, we show that such correlational sensors can also be used to selectively accept or reject light rays from certain scene depths. The basic idea is to carefully select illumination and sensor modulation patterns such that the correlation is non-zero only in the selected depth range -thus light reflected from objects outside this depth range do not affect the correlational measurements. We demonstrate a prototype depth-selective camera and highlight two potential applications: imaging through scattering media and virtual blue screening. This depthselectivity can be used to reject back-scattering and reflection from media in front of the subjects of interest, thereby significantly enhancing the ability to image through scattering media-critical for applications such as car navigation in fog and rain. Similarly, such depth selectivity can also be utilized as a virtual blue-screen in cinematography by rejecting light reflecting from background, while selectively retaining light contributions from the foreground subject.

Ellipsoidal path connections for time-gated rendering

“…CW-ToF depth-selective camera. We simulate a CW-ToF camera using modulation codes resulting in depth-selectivity [Tadano et al 2015].…”

“…Finally, even though most non-line-of-sight imaging algorithms require entire transients, some techniques use time-gated measurements [Laurenzis and Velten 2014;Pediredla et al 2019a;Thrampoulidis et al 2018] Time-gated imaging scenarios can arise even when using ToF technologies other than time-gated sensors. For example, even though typical CW-ToF sensors accumulate non-zero contributions from all photons regardless of their time-of-travel, it is possible to use special modulation patterns to create a camera that almost exclusively measures photons within a specific time-of-travel range [Tadano et al 2015]. Such cameras have been used for applications such as photography through partial occlusions, and optical Z-keying.…”

“…Typically, C is a sinusoid, with frequency in the range of tens to hundreds of MHz. However, cross-correlations of trapezoidal [Gupta et al 2018] and narrow rectangular [Tadano et al 2015] shapes have also been demonstrated. In the latter case, the camera effectively reduces to a time-gated camera for some narrow pathlength importance function W τ .…”

“…Then, each frame of the transient sequence is a time-gated measurement of an effectively different scene, and therefore temporal path reuse cannot be used. Finally, a third case is when rendering CW-ToF cameras where the crosscorrelation of illumination and sensor modulation has very narrow support, as in the depth-selective camera of Tadano et al [2015]. We use the term time-gated rendering to describe these and all other rendering tasks dealing with very narrow pathlength importance functions; these tasks will be the focus of the rest of the paper.…”

Towards human-like and transhuman perception in AI 2.0: a review

“…Following that, a high-resolution, short-waved infrared compressed sensing camera was developed in McMackin et al (2012), while a multi-view lenseless camera was developed in Jiang et al (2014), in which aberration and focusing problems associated with lenses could be completely avoided. In Kadambi et al (2013) and Tadano et al (2015), the authors aimed to develop a unified theory and practical designs for adaptive coded imaging and display, which could adapt to scene geometry, motion, and illumination to maximize the information throughput. Obviously, new computational imaging theory and technologies such as ultra-high speed imaging and light-field imaging are highly desirable due to some recent large technological demands such as automated driving and virtual reality.…”