RGB-D Local Implicit Function for Depth Completion of Transparent Objects

Zhu, Luyang; Mousavian, Arsalan; Yu, Xi; Mazhar, Hammad; Eenbergen, Jozef van; Debnath, Shoubhik; Fox, Dieter

doi:10.1109/cvpr46437.2021.00462

Cited by 52 publications

(48 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address depth smearing between objects, Imran et al [83] proposed a depth coefficient representation which enables convolutions to more easily avoid inter-object depth mixing. In recent work, Zhu et al [84] introduced a local implicit neural representation built on ray-voxel pairs that allows generalization to unseen transparent objects and provides fast inferencing.…”

Section: Data-driven Methodsmentioning

confidence: 99%

High-quality indoor scene 3D reconstruction with RGB-D cameras: A brief review

Gao

et al. 2022

Comp. Visual Media

View full text Add to dashboard Cite

High-quality 3D reconstruction is an important topic in computer graphics and computer vision with many applications, such as robotics and augmented reality. The advent of consumer RGB-D cameras has made a profound advance in indoor scene reconstruction. For the past few years, researchers have spent significant effort to develop algorithms to capture 3D models with RGB-D cameras. As depth images produced by consumer RGB-D cameras are noisy and incomplete when surfaces are shiny, bright, transparent, or far from the camera, obtaining high-quality 3D scene models is still a challenge for existing systems. We here review high-quality 3D indoor scene reconstruction methods using consumer RGB-D cameras. In this paper, we make comparisons and analyses from the following aspects: (i) depth processing methods in 3D reconstruction are reviewed in terms of enhancement and completion, (ii) ICP-based, feature-based, and hybrid methods of camera pose estimation methods are reviewed, and (iii) surface reconstruction methods are reviewed in terms of surface fusion, optimization, and completion. The performance of state-of-the-art methods is also compared and analyzed. This survey will be useful for researchers who want to follow best practices in designing new high-quality 3D reconstruction methods.

show abstract

Section: Data-driven Methodsmentioning

confidence: 99%

High-quality indoor scene 3D reconstruction with RGB-D cameras: A brief review

Gao

et al. 2022

Comp. Visual Media

View full text Add to dashboard Cite

show abstract

“…More sophisticated approaches include estimating bounding boxes based on the refraction distortions that transparent objects' edges create on the background [7]. Newer research trains deep convolutional neural networks on massive datasets to reach a pixel-wise labeling of transparent objects [2], [8]. Also, some researchers have devised clever methods to exploit sensor failures in depth images to localize transparent objects [9].…”

Section: Related Workmentioning

confidence: 99%

“…Regardless, often times researchers are forced to use low-quality real data for testing their algorithm or training a neural network [2], [7], and other times researchers resort to creating rigid virtual datasets to train large models, such as a DCNN. [2], [8], [9].…”

Section: Related Workmentioning

confidence: 99%

“…Modern computer graphics can achieve near-photorealism, so synthetic data has become a viable alternative in situations where acquiring or labeling real data is difficult. However, there are still few datasets that meet the image quality required to be considered photo-realistic [8], and those that do, rarely include transparent objects because of the complexities they introduce in rendering and generating correct ground-truth labels. More importantly, to the best of our knowledge, there is no photorealistic, synthetic dataset that includes the tools and project files needed to generate the data itself.…”

Section: Related Workmentioning

confidence: 99%

“…In older research, we see the use of personally-gathered real data that suffers from lack of variety, noise, or lack of labels. To get past this obstacle, we see various works generate their own synthetic datasets using tools like Blender [1], [2] or NVIDIA Omniverse [8], [15]. As a result, these datasets consist of huge collections of static images.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

SuperCaustics: Real-time, open-source simulation of transparent objects for deep learning applications

Mousavi¹,

Estrada²

2021

Preprint

View full text Add to dashboard Cite

Transparent objects are a very challenging problem in computer vision. They are hard to segment or classify due to their lack of precise boundaries, and there is limited data available for training deep neural networks. As such, current solutions for this problem employ rigid synthetic datasets, which lack flexibility and lead to severe performance degradation when deployed on real-world scenarios. In particular, these synthetic datasets omit features such as refraction, dispersion and caustics due to limitations in the rendering pipeline. To address this issue, we present SuperCaustics, a real-time, open-source simulation of transparent objects designed for deep learning applications. SuperCaustics features extensive modules for stochastic environment creation; uses hardware ray-tracing to support caustics, dispersion, and refraction; and enables generating massive datasets with multi-modal, pixel-perfect ground truth annotations. To validate our proposed system, we trained a deep neural network from scratch to segment transparent objects in difficult lighting scenarios. Our neural network achieved performance comparable to the state-of-the-art on a real-world dataset using only 10% of the training data and in a fraction of the training time. Further experiments show that a model trained with SuperCaustics can segment different types of caustics, even in images with multiple overlapping transparent objects. To the best of our knowledge, this is the first such result for a model trained on synthetic data. Both our open-source code and experimental data are freely available online.

show abstract

Context Dual-Branch Attention Network for Depth Completion of Transparent Object

Wang

Chen

et al. 2022

Intelligent Robotics and Applications

View full text Add to dashboard Cite

RGB-D Local Implicit Function for Depth Completion of Transparent Objects

Cited by 52 publications

References 34 publications

High-quality indoor scene 3D reconstruction with RGB-D cameras: A brief review

High-quality indoor scene 3D reconstruction with RGB-D cameras: A brief review

SuperCaustics: Real-time, open-source simulation of transparent objects for deep learning applications

Context Dual-Branch Attention Network for Depth Completion of Transparent Object

Contact Info

Product

Resources

About