Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge

Zeng, Andy; Yu, Kuan-Ting; Song, Shuran; Suo, Daniel; Walker, Ed; Rodríguez, Alberto; Xiao, Jianxiong

doi:10.48550/arxiv.1609.09475

Cited by 12 publications

(17 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An alternative approach is to use deep learning to estimate 3D object shape and pose directly from color and depth images [17,59]. Recent research in robotics has focused on how to improve accuracy in object recognition by structuring the way the neural network fuses the separate color and depth streams from images [43] and adding synthetic noise to synthetic training images [10].Another approach is to detect graspable regions directly in images without explicitly representing object shape and pose [31,38,40,50], as it may not always be necessary to explicitly recognize objects and their pose to perform a grasp.…”

Section: Related Workmentioning

confidence: 99%

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Mahler¹,

Liang²,

Niyaz³

et al. 2017

Robotics: Science and Systems XIII

938

936

View full text Add to dashboard Cite

Abstract-To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, DexNet 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8s with a success rate of 93% on eight known objects with adversarial geometry and is 3× faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net.

show abstract

Section: Related Workmentioning

confidence: 99%

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Mahler¹,

Liang²,

Niyaz³

et al. 2017

Robotics: Science and Systems XIII

938

936

View full text Add to dashboard Cite

show abstract

“…With the availability of powerful commodity GPUs, and fast detection al-gorithms [27,38], these methods are suitable for realtime object detection required in robotics. More recently, deep learning based approaches in computer vision are being adopted for the task of pose estimation of specific objects [33,53,54]. Improving instance detection and pose estimation in warehouses will be signifcantly useful for the perception pipeline in systems trying to solve the Amazon Picking Challenge [7].…”

Section: Related Workmentioning

confidence: 99%

Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection

Dwibedi

Misra

Hebert

2017

2017 IEEE International Conference on Computer Vision (ICCV)

532

476

View full text Add to dashboard Cite

A major impediment in rapidly deploying object detection models for instance detection is the lack of large annotated datasets. For example, finding a large labeled dataset containing instances in a particular kitchen is unlikely. Each new environment with new instances requires expensive data collection and annotation. In this paper, we propose a simple approach to generate large annotated instance datasets with minimal effort. Our key insight is that ensuring only patch-level realism provides enough training signal for current object detector models. We automatically 'cut' object instances and 'paste' them on random backgrounds. A naive way to do this results in pixel artifacts which result in poor performance for trained models. We show how to make detectors ignore these artifacts during training and generate data that gives competitive performance on real data. Our method outperforms existing synthesis approaches and when combined with real images improves relative performance by more than 21% on benchmark datasets. In a cross-domain setting, our synthetic data combined with just 10% real data outperforms models trained on all real data.

show abstract

“…Most DCNN-based detection and recognition methods are fully supervised, trained by massive annotated datasets. In contrast, weakly-supervised deep learning has, so far, only achieved success in very few applications, including path planing [20], and Amazon picking challenge [21]. In contrast, this paper shows how weakly supervised deep learning can achieve very strong performance on RGBD object detection and recognition, at real-time frame rates, on real-world industrial image data, for which only a tiny amount (0.3%) of labeled data is available for training.…”

Section: Discussionmentioning

confidence: 74%

“…In their method, the path in future frames are projected to the current frame through vehicle odometry and annotated as ground truth for learning. Zeng, et al [21] proposed a selfsupervised approach to learn fully-constitutional networks for object segmentation in the Amazon picking challenge.…”

Section: Weakly-supervised Deep Learningmentioning

confidence: 99%

Weakly-supervised DCNN for RGB-D Object Recognition in Real-World Applications Which Lack Large-scale Annotated Training Data

Sun,

Zhao,

Stolkin

2017

Preprint

View full text Add to dashboard Cite

This paper addresses the problem of RGBD object recognition in real-world applications, where large amounts of annotated training data are typically unavailable. To overcome this problem, we propose a novel, weakly-supervised learning architecture (DCNN-GPC) which combines parametric models (a pair of Deep Convolutional Neural Networks (DCNN) for RGB and D modalities) with non-parametric models (Gaussian Process Classification). Our system is initially trained using a small amount of labeled data, and then automatically propagates labels to large-scale unlabeled data. We first run 3Dbased objectness detection on RGBD videos to acquire many unlabeled object proposals, and then employ DCNN-GPC to label them. As a result, our multi-modal DCNN can be trained end-to-end using only a small amount of human annotation. Finally, our 3D-based objectness detection and multi-modal DCNN are integrated into a real-time detection and recognition pipeline. In our approach, bounding-box annotations are not required and boundary-aware detection is achieved. We also propose a novel way to pretrain a DCNN for the depth modality, by training on virtual depth images projected from CAD models. We pretrain our multi-modal DCNN on public 3D datasets, achieving performance comparable to state-of-the-art methods on Washington RGBS Dataset. We then finetune the network by further training on a small amount of annotated data from our novel dataset of industrial objects (nuclear waste simulants). Our weakly supervised approach has demonstrated to be highly effective in solving a novel RGBD object recognition application which lacks of human annotations.

show abstract

Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge

Cited by 12 publications

References 12 publications

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection

Weakly-supervised DCNN for RGB-D Object Recognition in Real-World Applications Which Lack Large-scale Annotated Training Data

Contact Info

Product

Resources

About