Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging

Chugunov, Ilya; Baek, Seung‐Hwan; Fu, Qiang; Heidrich, Wolfgang; Heide, Felix

doi:10.1109/cvpr46437.2021.00900

Cited by 8 publications

(5 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, depth cameras based on direct ToF -marketed as LiDAR scanners [78] -have been released on a few Apple smartphones, such as the Pro versions of iPhones 12 and 13 [4]. Direct ToF cameras use timed light pulses to gather depth measurements in a scene, instead of illuminating the whole scene at once with modulated light as with indirect ToF cameras [13]. Thus, the raw depth maps obtained from direct ToF cameras are sparse [31,48], and the task of depth completion is more suitable to completing sparse depth maps (see Section 2).…”

Section: Discussionmentioning

confidence: 99%

“…In the future, we will incorporate our depth inpainting technique into a full AR system with 6DoF tracking, and evaluate it with mobile AR users. Dynamic scenarios will also allow us to study the possible impact of motion blur in RGB images and flying-pixel artifacts in ToF depth images [13,98]. Incorporating InDepth into ARCore is an open challenge, however: although raw depth images can be acquired in ARCore [25], the Depth API does not currently support passing processed depth images back into the AR mapping and tracking algorithm.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

InDepth

Zhang

Scargill

Vaishnav

et al. 2022

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Mobile Augmented Reality (AR) demands realistic rendering of virtual content that seamlessly blends into the physical environment. For this reason, AR headsets and recent smartphones are increasingly equipped with Time-of-Flight (ToF) cameras to acquire depth maps of a scene in real-time. ToF cameras are cheap and fast, however, they suffer from several issues that affect the quality of depth data, ultimately hampering their use for mobile AR. Among them, scale errors of virtual objects - appearing much bigger or smaller than what they should be - are particularly noticeable and unpleasant. This article specifically addresses these challenges by proposing InDepth, a real-time depth inpainting system based on edge computing. InDepth employs a novel deep neural network (DNN) architecture to improve the accuracy of depth maps obtained from ToF cameras. The DNN fills holes and corrects artifacts in the depth maps with high accuracy and eight times lower inference time than the state of the art. An extensive performance evaluation in real settings shows that InDepth reduces the mean absolute error by a factor of four with respect to ARCore DepthLab. Finally, a user study reveals that InDepth is effective in rendering correctly-scaled virtual objects, outperforming DepthLab.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

InDepth

Zhang

Scargill

Vaishnav

et al. 2022

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

show abstract

“…an object, at which the depth abruptly changes. This may lead to an incorrect estimation of the depth in the incident pixel [76], [77]. To evaluate this phenomenon, we make use of the 1 × 1 m white panel.…”

Section: F Key Parameters For a Tof Cameramentioning

confidence: 99%

Performance Evaluation of State-of-the-Art High-Resolution Time-of-Flight Cameras

2023

View full text Add to dashboard Cite

It has been about 20 years since the disruptive appearance of the first time-of-flight (ToF) cameras. Since then, ToF imaging has progressively evolved. Nowadays, ToF sensors have broken the barrier of the 1-megapixel resolution, and a significant number of high-resolution ToF cameras have appeared in the market. To provide a better understanding of their performance and applications, we experimentally evaluate three state-of-the-art high-resolution ToF cameras such as Azure Kinect, Helios2, and S100D, together with the solid-state LiDAR L515. We perform various experiments to examine some key parameters, such as warmup times, accuracy, precision, lateral and axial resolutions, edge noise, unsteady scenes, and modulated waveform and optical power. Our evaluation draws various conclusions: S100D shows fluctuations within 1 mm after being powered up while the others require warm-up times. Azure Kinect, Helios2, and L515 can achieve precision within 2 mm in a measuring range of 0.5-3 m. Helios2 and S100D are more severely affected by dynamic scenes. Finally, the point clouds (PCs) generated for a white panel at a distance of 1.5 m show that flying pixels are present in all cameras, being this problem less acute for the L515.

show abstract

“…Conventional imaging systems are typically designed in a sequential approach, where lens and sensors are hand-engineered based on specific metrics, such as PSF spot size or dynamic range, independent of the downstream camera task. Departing from this conventional design approach, a large body of work in computational imaging has explored jointly optimizing the optics and reconstruction algorithms, with successful applications in color image restoration [Chakrabarti 2016;, microscopy [Horstmeyer et al 2017;Kellman et al 2019;Nehme et al 2020;Shechtman et al 2016], monocular depth imaging [Chang and Wetzstein 2019;Haim et al 2018;He et al 2018;, super-resolution and extended depth of field [Sitzmann et al 2018;Sun et al 2021], time-of-flight imaging [Chugunov et al 2021;Marco et al 2017;, high-dynamic range imaging [Metzler et al 2019;Sun et al 2020], active-stereo imaging [Baek and Heide 2021], hyperspectral imaging , and other computer vision tasks [Tseng et al 2021b].…”

Section: Related Workmentioning

confidence: 99%

Seeing through obstructions with diffractive cloaking

et al. 2022

Self Cite

View full text Add to dashboard Cite

Unwanted camera obstruction can severely degrade captured images, including both scene occluders near the camera and partial occlusions of the camera cover glass. Such occlusions can cause catastrophic failures for various scene understanding tasks such as semantic segmentation, object detection, and depth estimation. Existing camera arrays capture multiple redundant views of a scene to see around thin occlusions. Such multi-camera systems effectively form a large synthetic aperture, which can suppress nearby occluders with a large defocus blur, but significantly increase the overall form factor of the imaging setup. In this work, we propose a monocular single-shot imaging approach that optically cloaks obstructions by emulating a large array. Instead of relying on different camera views, we learn a diffractive optical element (DOE) that performs depth-dependent optical encoding, scattering nearby occlusions while allowing paraxial wavefronts to be focused. We computationally reconstruct unobstructed images from these superposed measurements with a neural network that is trained jointly with the optical layer of the proposed imaging system. We assess the proposed method in simulation and with an experimental prototype, validating that the proposed computational camera is capable of recovering occluded scene information in the presence of severe camera obstruction.

show abstract

Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging

Cited by 8 publications

References 58 publications

InDepth

InDepth

Performance Evaluation of State-of-the-Art High-Resolution Time-of-Flight Cameras

Seeing through obstructions with diffractive cloaking

Contact Info

Product

Resources

About