Multi-view self-supervised deep learning for 6D pose estimation in the Amazon Picking Challenge

Abstract

Robot warehouse automation has attracted significant interest in recent years, perhaps most visibly in the Amazon Picking Challenge (APC) [1]. A fully autonomous warehouse pick-and-place system requires robust vision that reliably recognizes and locates objects amid cluttered environments, self-occlusions, sensor noise, and a large variety of objects. In this paper we present an approach that leverages multiview RGB-D data and self-supervised, data-driven learning to overcome those difficulties. The approach was part of the MIT-Princeton Team system that took 3rd-and 4th-place in the stowing and picking tasks, respectively at APC 2016.In the proposed approach, we segment and label multiple views of a scene with a fully convolutional neural network, and then fit pre-scanned 3D object models to the resulting segmentation to get the 6D object pose. Training a deep neural network for segmentation typically requires a large amount of training data. We propose a self-supervised method to generate a large labeled dataset without tedious manual segmentation. We demonstrate that our system can reliably estimate the 6D pose of objects under a variety of scenarios. All code, data, and benchmarks are available at

Hand-Object Interaction Detection with Fully Convolutional Networks

“…Conversely, in the third example (Figure 11, third column), the left hand and object are correctly predicted to be separated. We also found that hands and objects can be detected in isolation, which reaffirms similar observations made in previous works [7,28].…”

“…These are then used in a complex multi-stage classification scheme for offline action recognition, while our approach uses the FCN outputs to discriminate the hand from the object and to generate pixel labels for real-time tracking. Similar to our approach, [28] used FCNs to discriminate between objects for 6D pose estimation. While this approach produces good results for object localization in cluttered environments, it requires a multicamera setup and does not achieve real-time performance, which is crucial for dynamic hand-object interactions.…”

Cost Aware Offloading Selection and Resource Allocation for Cloud Based Multi-Robot Systems

“…Taking an object pose estimation task [10] as an example, the input data includes a set of images and the total size of it could be hundreds of kilobytes at least, while the output data is just the object location and pose, and takes a few dozens of bytes at most.…”

Probabilistic 3D multilabel real-time mapping for multi-object manipulation

“…In addition to these works on 2D segmentation, three-dimentional segmentation is required for robot to conduct tasks in the real world. In order to achieve this, previous works propose projection-based approach projecting segmented pixels to 3D points in a single view (2.5D) [9], mapping-based approach with binary object existence [12] and probabilistic existence [1] for a single target object. And as for fully 3D-based approach, model matching is tackled [13] [14] using various 3D features [15] [16].…”