Fusing Depth and Silhouette for Scanning Transparent Object with RGB-D Sensor

Ji, Yong Gu; Qin, Xia; Zhang, Zhijiang

doi:10.1155/2017/9796127

Cited by 20 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, while our system is able to handle partially transparent kits, it has trouble handling fully transparent ones like the deodorant blister pack (we spray-paint it to support stereo matching for our 3D camera). Exploring the use of external vision algorithms like [33], [34], [35], [36] to estimate the geometry of the transparent kits before using the visual data would be a promising direction for future research.…”

Section: Discussion and Future Workmentioning

confidence: 99%

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

Zakka

Zeng

Lee

et al. 2019

Preprint

View full text Add to dashboard Cite

Is it possible to learn policies for robotic assembly that can generalize to new objects? We explore this idea in the context of the kit assembly task. Since classic methods rely heavily on object pose estimation, they often struggle to generalize to new objects without 3D CAD models or taskspecific training data. In this work, we propose to formulate the kit assembly task as a shape matching problem, where the goal is to learn a shape descriptor that establishes geometric correspondences between object surfaces and their target placement locations from visual input. This formulation enables the model to acquire a broader understanding of how shapes and surfaces fit together for assembly -allowing it to generalize to new objects and kits. To obtain training data for our model, we present a self-supervised data-collection pipeline that obtains ground truth object-to-placement correspondences by disassembling complete kits. Our resulting real-world system, Form2Fit, learns effective pick and place strategies for assembling objects into a variety of kits -achieving 90% average success rates under different initial conditions (e.g. varying object and kit poses), 94% success under new configurations of multiple kits, and over 86% success with completely new objects and kits. Code, videos, and supplemental material are available at https://form2fit.github.io 1

show abstract

Section: Discussion and Future Workmentioning

confidence: 99%

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

Zakka

Zeng

Lee

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Works in this area generally fall in four main schemes: constraints driven, monocular depth estimation, depth completion from sparse point cloud, and depth completion given noisy RGB-D. Our work falls in the last scheme. Constraint driven approaches assume a specific setup method with fixed viewpoint(s) and capturing procedure [12,13,14,15,16], sensor type [17,18,11], or known object models [19,20,21]. Our proposed depth completion method does not apply any assumptions.…”

Section: Related Workmentioning

confidence: 99%

Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects

Xu,

Wang,

Eppel

et al. 2021

Preprint

View full text Add to dashboard Cite

The basis of many object manipulation algorithms is RGB-D input. Yet, commodity RGB-D sensors can only provide distorted depth maps for a wide range of transparent objects due light refraction and absorption. To tackle the perception challenges posed by transparent objects, we propose TranspareNet, a joint point cloud and depth completion method, with the ability to complete the depth of transparent objects in cluttered and complex scenes, even with partially filled fluid contents within the vessels. To address the shortcomings of existing transparent object data collection schemes in literature, we also propose an automated dataset creation workflow that consists of robot-controlled image collection and vision-based automatic annotation. Through this automated workflow, we created Toronto Transparent Objects Depth Dataset (TODD), which consists of nearly 15000 RGB-D images. Our experimental evaluation demonstrates that TranspareNet outperforms existing state-of-the-art depth completion methods on multiple datasets, including Clear-Grasp, and that it also handles cluttered scenes when trained on TODD. Code and dataset will be released at https://www.pair.toronto.edu/TranspareNet/

show abstract

“…3D geometry can be reconstructed from multiple views or 3D scanning. Ji et al [37] conducted a volumetric reconstruction of transparent objects by fusing depth and silhouette from multiple images with known poses. Li et al [38] presented a physically based network to recover 3D shape of transparent object from multiple color images.…”

Section: Related Workmentioning

confidence: 99%

6DoF Pose Estimation of Transparent Object from a Single RGB-D Image

Chen

Yao

et al. 2020

Sensors

View full text Add to dashboard Cite

6DoF object pose estimation is a foundation for many important applications, such as robotic grasping, automatic driving, and so on. However, it is very challenging to estimate 6DoF pose of transparent object which is commonly seen in our daily life, because the optical characteristics of transparent material lead to significant depth error which results in false estimation. To solve this problem, a two-stage approach is proposed to estimate 6DoF pose of transparent object from a single RGB-D image. In the first stage, the influence of the depth error is eliminated by transparent segmentation, surface normal recovering, and RANSAC plane estimation. In the second stage, an extended point-cloud representation is presented to accurately and efficiently estimate object pose. As far as we know, it is the first deep learning based approach which focuses on 6DoF pose estimation of transparent objects from a single RGB-D image. Experimental results show that the proposed approach can effectively estimate 6DoF pose of transparent object, and it out-performs the state-of-the-art baselines by a large margin.

show abstract

Fusing Depth and Silhouette for Scanning Transparent Object with RGB-D Sensor

Cited by 20 publications

References 24 publications

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly

Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects

6DoF Pose Estimation of Transparent Object from a Single RGB-D Image

Contact Info

Product

Resources

About