Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects

Abstract

Commercial depth sensors usually generate noisy and missing depths, especially on specular and transparent objects, which poses critical issues to downstream depth or point cloud-based tasks. To mitigate this problem, we propose a powerful RGBD fusion network, Swin-DRNet, for depth restoration. We further propose Domain Randomization-Enhanced Depth Simulation (DREDS) approach to simulate an active stereo depth system using physically based rendering and generate a large-scale synthetic dataset that contains 130K photorealistic RGB images along with their simulated depths carrying realistic sensor noises. To evaluate depth restoration methods, we also curate a real-world dataset, namely STD, that captures 30 cluttered scenes composed of 50 objects with different materials from specular, transparent, to diffuse. Experiments demonstrate that the proposed DREDS dataset bridges the sim-to-real domain gap such that, trained on DREDS, our SwinDRNet can seamlessly generalize to other real depth datasets, e.g. ClearGrasp, and outperform the competing methods on depth restoration with a real-time speed. We further show that our depth restoration effectively boosts the performance of downstream tasks, including category-level pose estimation and grasping tasks. Our data and code are available at https://github.com/PKU-EPIC/DREDS.

TransNet: Category-Level Transparent Object Pose Estimation

“…ClearGrasp [1] employed depth completion for use with pose estimation on robotic grasping tasks, where they trained three DeepLabv3+ [19] models to learn transparency mask, surface normal, and boundary, respectively. Follow-on studies developed different approaches for depth completion, including implicit functions [20], NeRF reconstruction [2], combined point cloud and depth features [14], adversarial learning [21], multi-view geometry [22], RGB image completion [3], and sim2real transfer [6]. Weng et al [4] used transfer learning from the RGB to the depth sensor domain without completing raw depth.…”

“…However, these techniques require high-quality depth input provided by opaque objects with Lambertian light reflectance. One recent work by Dai et al [6] proposed a data generation system that simulates the noise on non-Lambertian surfaces of active stereo depth cameras and demonstrated its usage in category-level pose estimation and robotic grasping.…”

“…Recent works have shown promising results on grasping transparent objects by completing the missing depth values followed by the use of a geometry-based grasp engine [1], [2], [3], transfer learning from RGB-based grasping neural networks [4], light-field feature learning [5], or domainrandomized depth noise simulation [6]. For more advanced manipulation tasks such as rigid body pick-and-place or liquid pouring, geometry-based estimations, such as symmetrical axes, edges [7] or object poses [8], [9], [6], are required to model the manipulation trajectories. Instance-level transparent object poses could be estimated from keypoints on stereo RGB images [10], [11], a light-field camera [12], [13], or directly from a single RGB-D image [9] with support plane assumptions.…”