Robot grasp detection using multimodal deep convolutional neural networks

Abstract

Autonomous manipulation has enabled a wide range of exciting robot tasks. However, perceiving outside environment is still a challenging problem in the field of intelligent robotic research due to the lack of object models, unstructured environments, and time-consuming computation. In this article, we present a novel robot grasp detection system that maps a pair of RGB-D images of novel objects to best grasping pose of a robotic gripper. First, we segment the graspable objects from the unstructured scene using the geometrical features of both the object and the robotic gripper. Then, a deep convolutional neural network is applied on these graspable objects, which aims to find the best graspable area for each object. In order to improve the efficiency in the detection system, we introduce a structured penalty term to optimize the connections between multimodality, which significantly mitigates complexity of the network and outperforms fully connected multimodal processing. We also present a two-stage closed-loop grasping candidate estimator to accelerate the searching efficiency of grasping-candidate generation. Moreover, the combination of a two-stage estimator with the grasping detection network naturally improves detection accuracy. Experiments have been conducted to validate the proposed methods. The results show that our method outperforms the state of the art and runs at real-time speed.

Review of Deep Learning Methods in Robotic Grasp Detection

“…In their novel classification method for grasp detection, Zhou et al [34] used a similar five-element grasp rectangle representation following the previous work in [10,[18][19][20][31][32][33]35]. Wang et al [36] proposed a minor variation to this approach that differed simply by excluding the parameter for gripper plate height (h). They argued that this parameter can be controlled in the robotic set-up configurations thus the authors used a four-element grasp representation of G = (x, y, θ, w).…”

“…In the grasp detection work by Wang et al [36], the authors used the Washington RGB-D dataset [58] for its rich variety of RGB-D images. The authors self-annotated as they preferred to combine the resulting dataset with the CGD.…”

“…The authors self-annotated as they preferred to combine the resulting dataset with the CGD. The authors further stated that the combined Washington data instances of 25,000 with the 885 instances from the CGD would help in pre-training a deep network [36]. …”

Dealing with Ambiguity in Robotic Grasping via Multiple Predictions

“…A shallow network first predicts high-ranked candidate grasp rectangles, followed by a deeper network that chooses the optimal grasp points. Wang et al [38] followed a similar approach using a multi-modal CNN. Another method [15] uses RGB-D data to first extract features from a scene using a ResNet-50 architecture [11] and then a successive shallower convolutional network applied to the merged features to estimate the optimal point of grasping.…”

“…This tends to slow down the overall run-time and fails in presence of complicated or unseen object shapes. Following the success of deep learning in a wide spectrum of computer vision applications, several recent approaches [9,15,20,24,29,37,38] employed Convolutional Neural Networks (CNNs) [14,18] to successfully detect grasping points from visual data, typically parametrized by 5dimensional (5D) grasping representations [12,20]. It is worth noting that most of these methods rely on depth data, often paired with color information.…”