Visual Manipulation Relationship Network for Autonomous Robotics

Abstract

Robotic grasping is one of the most important fields in robotics and convolutional neural network (CNN) has made great progress in detecting robotic grasps. However, including multiple objects in one scene can invalidate the existing grasping detection algorithms based on CNN because of lacking of manipulation relationships among objects to guide the robot to grasp things in the right order. Therefore, the manipulation relationships are needed to help robot better grasp and manipulate objects. This paper presents a new CNN architecture called Visual Manipulation Relationship Network (VMRN) to help robot detect targets and predict the manipulation relationships in real time. To implement end-to-end training and meet real-time requirements in robot tasks, we propose the Object Pairing Pooling Layer (OP 2 L) to help to predict all manipulation relationships in one forward process. In order to train VMRN, we collect a dataset named Visual Manipulation Relationship Dataset (VMRD) consisting of 5185 images with more than 17000 object instances and the manipulation relationships between all possible pairs of objects in every image, which is labeled by the manipulation relationship tree. The experiment results show that the new network architecture can detect objects and predict manipulation relationships simultaneously and meet the real-time requirements in robot tasks.

A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes

“…We can see that our model improves the performance of visual manipulation relationship reasoning compared with our previous work [6]. We assume that the improvements are achieved following these changes: (1) Different from our previous work, in this paper, we use ResNet-101 as the feature extractor, or called "backbone" instead of ResNet-50 and VGG-16 in [6]; (2) the object detector is Faster-RCNN from [18] instead of SSD in [19]; (3) the backbone is updated using multi-task loss function including grasp detection loss.…”

“…Recent works prove that CNNs achieve advanced performance on visual relationship reasoning [14]- [16]. Different from visual relationship, visual manipulation relationship [6] is proposed to solve the problem of grasping order in object stacking scenes with consideration of the safety and stability of objects. However, when this algorithm is directly combined with the grasp detection network to solve grasping problem in object stacking scenes, there are two main difficulties: 1) it is difficult to correctly match the detected grasps and the detected objects in object stacking scenes; 2) the cascade structure causes a lot of redundant calculations (e.g.…”

“…Similarly, reasoning metrics are used to test the performance of the reasoning output on the test set of VMRD. This part will take our previous work [6] as the baseline. Grasping metrics are used to evaluate how well the proposed network performs in real-world experiments.…”

“…Though there are some previous works that try to complete grasping in dense clutter [2]- [5], as we know, our proposed algorithm is the first to combine perception, reasoning, and grasp planning simultaneously by using one neural network, and attempts to realize autonomous robotic grasp in complex scenarios. To evaluate our proposed algorithm, we validate the performance of our model in VMRD dataset [6]. For robotic experiments, Baxter robot is used as the executor to complete grasping tasks, in which the robot is required to find the target, make the plan for grasping and grasp the target step by step.…”

Keypoint-Based Robotic Grasp Detection Scheme in Multi-Object Scenes

“…On the Cornell dataset, the accuracy of our scheme in image splitting and object splitting is 97.12% and 95.89%, respectively, which is equivalent to the most advanced grasping detection algorithm. In order to verify the effect of the scheme in multi-object scenarios, we used the VMRD dataset [ 14 ] containing multi-object scenarios, which produced an accuracy of 74.3%. As shown in Figure 1 , our method is also applied to real robot grasping tasks.…”