Learning Depth Completion of Transparent Objects using Augmented Unpaired Data

Erich, Floris; Leme, Bruno; Ando, Noriaki; Hanai, Ryo; Domae, Yukiyasu

doi:10.1109/icra48891.2023.10160619

Cited by 3 publications

(3 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(3) Applications to create a real (stereo) training dataset for deep neural networks using our new TranSpec3D dataset of transparent and specular objects without object painting (cf. [ 36 , 40 , 41 ]).…”

Section: Conclusion Limitations and Future Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…In the field of deep stereo, for example, the generation of real training data with dense ground truth disparities is very complex (costly and time-consuming), especially for visually uncooperative objects in the visible spectral range [ 1 , 36 , 40 , 41 ], e.g., specular, non-reflective, or non-textured surfaces. Here, the painting of uncooperative objects is SOTA [ 36 , 40 , 41 ]. However, in collaboration with the Fraunhofer Institute IOF, we have developed a new method without object painting based on a novel 3D thermal infrared sensing technique by Landmann et al [ 42 ].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Junger

Buch

Notni

2023

Sensors

View full text Add to dashboard Cite

The projection of a point cloud onto a 2D camera image is relevant in the case of various image analysis and enhancement tasks, e.g., (i) in multimodal image processing for data fusion, (ii) in robotic applications and in scene analysis, and (iii) for deep neural networks to generate real datasets with ground truth. The challenges of the current single-shot projection methods, such as simple state-of-the-art projection, conventional, polygon, and deep learning-based upsampling methods or closed source SDK functions of low-cost depth cameras, have been identified. We developed a new way to project point clouds onto a dense, accurate 2D raster image, called Triangle-Mesh-Rasterization-Projection (TMRP). The only gaps that the 2D image still contains with our method are valid gaps that result from the physical limits of the capturing cameras. Dense accuracy is achieved by simultaneously using the 2D neighborhood information (rx,ry) of the 3D coordinates in addition to the points P(X,Y,V). In this way, a fast triangulation interpolation can be performed. The interpolation weights are determined using sub-triangles. Compared to single-shot methods, our algorithm is able to solve the following challenges. This means that: (1) no false gaps or false neighborhoods are generated, (2) the density is XYZ independent, and (3) ambiguities are eliminated. Our TMRP method is also open source, freely available on GitHub, and can be applied to almost any sensor or modality. We also demonstrate the usefulness of our method with four use cases by using the KITTI-2012 dataset or sensors with different modalities. Our goal is to improve recognition tasks and processing optimization in the perception of transparent objects for robotic manufacturing processes.

show abstract

Section: Conclusion Limitations and Future Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Junger

Buch

Notni

2023

Sensors

View full text Add to dashboard Cite

show abstract

Scanning and Affordance Segmentation of Glass and Plastic Bottles

Erich,

Ando,

Yoshiyasu

2024

2024 IEEE/SICE International Symposium on System Integration (SII)

View full text Add to dashboard Cite

TranSpec3D: A Novel Measurement Principle to Generate A Non-Synthetic Data Set of Transparent and Specular Surfaces without Object Preparation

Junger,

Speck,

Landmann

et al. 2023

Sensors

View full text Add to dashboard Cite

Estimating depth from images is a common technique in 3D perception. However, dealing with non-Lambertian materials, e.g., transparent or specular, is still nowadays an open challenge. However, to overcome this challenge with deep stereo matching networks or monocular depth estimation, data sets with non-Lambertian objects are mandatory. Currently, only few real-world data sets are available. This is due to the high effort and time-consuming process of generating these data sets with ground truth. Currently, transparent objects must be prepared, e.g., painted or powdered, or an opaque twin of the non-Lambertian object is needed. This makes data acquisition very time consuming and elaborate. We present a new measurement principle for how to generate a real data set of transparent and specular surfaces without object preparation techniques, which greatly reduces the effort and time required for data collection. For this purpose, we use a thermal 3D sensor as a reference system, which allows the 3D detection of transparent and reflective surfaces without object preparation. In addition, we publish the first-ever real stereo data set, called TranSpec3D, where ground truth disparities without object preparation were generated using this measurement principle. The data set contains 110 objects and consists of 148 scenes, each taken in different lighting environments, which increases the size of the data set and creates different reflections on the surface. We also show the advantages and disadvantages of our measurement principle and data set compared to the Booster data set (generated with object preparation), as well as the current limitations of our novel method.

show abstract

Learning Depth Completion of Transparent Objects using Augmented Unpaired Data

Cited by 3 publications

References 31 publications

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Triangle-Mesh-Rasterization-Projection (TMRP): An Algorithm to Project a Point Cloud onto a Consistent, Dense and Accurate 2D Raster Image

Scanning and Affordance Segmentation of Glass and Plastic Bottles

TranSpec3D: A Novel Measurement Principle to Generate A Non-Synthetic Data Set of Transparent and Specular Surfaces without Object Preparation

Contact Info

Product

Resources

About