Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data

Abstract

The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive handlabeled datasets are available. As generating these datasets is time-consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can transfer successfully to the real world. We present a method for automated dataset generation and rapidly generate a synthetic training dataset of 50,000 depth images and 320,000 object masks using simulated heaps of 3D CAD models. We train a variant of Mask R-CNN with domain randomization on the generated dataset to perform category-agnostic instance segmentation without any handlabeled data and we evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution depth images of challenging, densely-cluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall on COCO benchmarks, and achieves performance levels similar to a Mask R-CNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE. arXiv:1809.05825v2 [cs.CV]

A Survey on Learning-Based Robotic Grasping

“…This technique has been successfully used for object localization [48••], segmentation [74], robot control for pick-andplace [75], swing-peg-in-hole [76], opening a cabinet drawer [76], in-hand manipulation [77], one-handed Rubik's Cube solving [78], precise 6D pose regression in highly cluttered environments [20•], etc. Modifications propose an automatic scheduling of the intensity of the randomization based on the current performance of the system [78] or adapting simulation randomizations by using real-world data to identify distributions that are particularly suited for a successful transfer [76].…”

Convolutional Neural Networks–Based Image Analysis for the Detection and Quantification of Neutrophil Extracellular Traps

“…For our work, we chose an open source implementation by Matterport [34] in Python with the use of Keras and Tensorflow frameworks. The model and its implementation has already gained popularity among various researchers [35][36][37][38][39][40] for of several reasons-The model is published under MIT license, which allows users to modify the model; it adopts well established CNN backbone ResNet [41] and recently introduced concepts like Feature Pyramid Network (FPN) [42] and ROI Align that in terms of quality make Mask R-CNN superior to comparable models like Faster R-CNN; the maximum accepted input image resolution of the model (1024 × 1024 pixels) is high when compared to many previously developed CNN models like YOLO (up to 608 × 608 pixels) [43] or Faster R-CNN [42] Python implementation (600 × 1000 pixels). The ability to analyze higher resolution images is important especially when dealing with small objects like biological cells [35].…”

Marker-Less 3d Object Recognition and 6d Pose Estimation for Homogeneous Textureless Objects: An RGB-D Approach

“…In case the objects do not have colour or textural information, this approach fails to produce reliable results. More recently, people used different Artificial Neural Networks (ANN) for 3d object recognition and 6d pose estimation [ 18 , 19 , 28 , 29 ]. Gupta et al [ 18 ] used both colour images and depth features to train a Convolutional Neural Network (CNN) model.…”

“…To improve the time performance of shape retrieval approaches, researchers [ 16 ] suggested to move the heavy computation processes to the offline stages. With the affordable price and accessibility of the RGB-D sensors, researchers proposed different object detection and pose estimation methods using both optical and depth information [ 10 , 17 , 18 , 19 , 20 ]. Although these methods usually outperform the approaches based on optical information only, depth sensors have a limited capturing angle and are more sensitive to illumination conditions.…”