Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset

Guo, Qi; Frosio, Iuri; Gallo, Orazio; Zickler, Todd; Kautz, Jan

doi:10.1007/978-3-030-01246-5_23

Cited by 48 publications

(70 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…represents a key advancement of this work w.r.t. previous research on the topic [5,14,30,24]. The discriminator is fed with a two channel input.…”

Section: Proposed Methodsmentioning

confidence: 99%

“…Recently, deep learning techniques using end-to-end CNNs, taking raw ToF correlation samples as input and outputting the refined scene depth map, have been presented for general purpose ToF denoising [30,14,5]. In these methods, the CNNs have been trained on synthetic data, but the adaptation to real data has been investigated only from a qualitative point of view in [30] and on a single corner scene in [14]. In the method of Agresti et al [5], a CNN for MPI correction is trained on multi-frequency synthetic data.…”

Section: Related Workmentioning

confidence: 99%

“…However, traditional approaches based on this idea do not have completely satisfactory performances. Recently, machine learning techniques and in particular Convolutional Neural Networks (CNN) have been exploited for this task [5,30,14]. Even if these approaches have obtained reasonable performances in some simple situations, they are not able to completely remove the MPI corruption and their generalization capabilities are limited.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning

Agresti

Schaefer

Sartor

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Time-of-Flight data is typically affected by a high level of noise and by artifacts due to Multi-Path Interference (MPI). While various traditional approaches for ToF data improvement have been proposed, machine learning techniques have seldom been applied to this task, mostly due to the limited availability of real world training data with depth ground truth. In this paper, we avoid to rely on labeled real data in the learning framework. A Coarse-Fine CNN, able to exploit multi-frequency ToF data for MPI correction, is trained on synthetic data with ground truth in a supervised way. In parallel, an adversarial learning strategy, based on the Generative Adversarial Networks (GAN) framework, is used to perform an unsupervised pixel-level domain adaptation from synthetic to real world data, exploiting unlabeled real world acquisitions. Experimental results demonstrate that the proposed approach is able to effectively denoise real world data and to outperform stateof-the-art techniques.

show abstract

“…represents a key advancement of this work w.r.t. previous research on the topic [5,14,30,24]. The discriminator is fed with a two channel input.…”

Section: Proposed Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning

Agresti

Schaefer

Sartor

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…MPI is caused not just by light scattering in participating media but also by subsurface scattering or interreflection in common scenes. Thus, many previous studies have tackled MPI compensation [24]- [28].…”

Section: Multipath Interference Of Tofmentioning

confidence: 99%

Simultaneous Estimation of Object Region and Depth in Participating Media Using a ToF Camera

Fujimura

Sonogashira

Iiyama

2020

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Three-dimensional (3D) reconstruction and scene depth estimation from 2-dimensional (2D) images are major tasks in computer vision. However, using conventional 3D reconstruction techniques gets challenging in participating media such as murky water, fog, or smoke. We have developed a method that uses a continuous-wave time-of-flight (ToF) camera to estimate an object region and depth in participating media simultaneously. The scattered light observed by the camera is saturated, so it does not depend on the scene depth. In addition, received signals bouncing off distant points are negligible due to light attenuation, and thus the observation of such a point contains only a scattering component. These phenomena enable us to estimate the scattering component in an object region from a background that only contains the scattering component. The problem is formulated as robust estimation where the object region is regarded as outliers, and it enables the simultaneous estimation of an object region and depth on the basis of an iteratively reweighted least squares (IRLS) optimization scheme. We demonstrate the effectiveness of the proposed method using captured images from a ToF camera in real foggy scenes and evaluate the applicability with synthesized data.

show abstract

“…. $15.00 DOI: 10.1145/3306214.3338582 us to synthesize reliable time-resolved light transport, and have already been proved useful in data-driven methods that address transient imaging problems [Guo et al 2018;]. However, the increased dimensionality of time-resolved renders dramatically boosts the size of the data, proportional to the resolution of the temporal dimension.…”

Section: Introductionmentioning

confidence: 99%

A data-driven compression method for transient rendering

Liang

Chen

Huang

et al. 2019

ACM SIGGRAPH 2019 Posters

View full text Add to dashboard Cite

Monte Carlo methods for transient rendering have become a powerful instrument to generate reliable data in transient imaging applications, either for benchmarking, analysis, or as a source for data-driven approaches. However, due to the increased dimensionality of time-resolved renders, storage and data bandwidth are signi cant limiting constraints, where a single time-resolved render of a scene can take several hundreds of megabytes. In this work we propose a learning-based approach that makes use of deep encoderdecoder architectures to learn lower-dimensional feature vectors of time-resolved pixels. We demonstrate how our method is capable of compressing transient renders up to a factor of 32, and recover the full transient pro le making use of a decoder. Additionally, we show how our learned features signi cantly mitigate variance on the recovered signal, addressing one of the pathological problems in transient rendering. CCS CONCEPTS •Computing methodologies → Ray tracing;

show abstract

Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset

Cited by 48 publications

References 24 publications

Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning

Unsupervised Domain Adaptation for ToF Data Denoising With Adversarial Learning

Simultaneous Estimation of Object Region and Depth in Participating Media Using a ToF Camera

A data-driven compression method for transient rendering

Contact Info

Product

Resources

About