6DoF Pose Estimation of Transparent Object from a Single RGB-D Image

Abstract

6DoF object pose estimation is a foundation for many important applications, such as robotic grasping, automatic driving, and so on. However, it is very challenging to estimate 6DoF pose of transparent object which is commonly seen in our daily life, because the optical characteristics of transparent material lead to significant depth error which results in false estimation. To solve this problem, a two-stage approach is proposed to estimate 6DoF pose of transparent object from a single RGB-D image. In the first stage, the influence of the depth error is eliminated by transparent segmentation, surface normal recovering, and RANSAC plane estimation. In the second stage, an extended point-cloud representation is presented to accurately and efficiently estimate object pose. As far as we know, it is the first deep learning based approach which focuses on 6DoF pose estimation of transparent objects from a single RGB-D image. Experimental results show that the proposed approach can effectively estimate 6DoF pose of transparent object, and it out-performs the state-of-the-art baselines by a large margin.

Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping

“…Rather than reconstructing a depth map, [4], [15] generate the grasp proposal with only RGB images or noisy depth maps as input. In [4], transfer learning was used to transfer the grasping model trained on depth maps to transparent object grasping with RGB images.…”

“…In [4], transfer learning was used to transfer the grasping model trained on depth maps to transparent object grasping with RGB images. In [15], a two-stage approach was proposed to estimate 6-DoF pose of transparent objects from a single RGB-D image, which can be used to assist transparent object grasping. Nonetheless, there has been no work on transparent object grasping with both visual and tactile information.…”

Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects

“…Phillips et al [8] trained a random forest to detect the contours of transparent objects for the purpose of pose estimation and shape recovery. Xu et al [9] proposed a two-stage method for estimating the 6-degrees-of-freedom (DOF) pose of a transparent object with a single RGBD image by replacing the noisy depth values with estimated values and training a DenseFusion-like network structure [10] to predict the object's 6-DOF pose. Sajjan et al [11] extend this and incorporate a neural network trained for 3D pose estimation of transparent objects in a robotic picking pipeline, while Zhou et al [12,13] train a grasp planner directly on raw images from a light-field camera.…”

“…Data-driven approaches learn a prior using labeled data [25,26] or through self-supervision over many trials in a simulated or physical environment [27,28] and generalize to grasping novel objects with unknown geometry. Both approaches rely on RGB and depth sensors to generate a sufficiently accurate observation of the target object surface, such as depth maps [29,30,31], point clouds [32,33,34,9], octrees [35], or a truncated signed distance function (TSDF) [36,37] from which it can compute the grasp pose. While various grasp-planning methods use different input geometry to compute grasps, in this paper we propose a method to render a high-quality depth map from a NeRF model to then pass to Dex-Net [29] to compute a grasp.…”

Seeing Glass: Joint Point Cloud and Depth Completion for Transparent Objects

“…Works in this area generally fall in four main schemes: constraints driven, monocular depth estimation, depth completion from sparse point cloud, and depth completion given noisy RGB-D. Our work falls in the last scheme. Constraint driven approaches assume a specific setup method with fixed viewpoint(s) and capturing procedure [12,13,14,15,16], sensor type [17,18,11], or known object models [19,20,21]. Our proposed depth completion method does not apply any assumptions.…”