Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning

Abstract

Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors amid challenging cases of clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after only a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at

A Deep Learning Approach to Grasping the Invisible

“…The goal of the experiments are: 1) to show the importance of the extra domain knowledge in the actors, 2) to understand the limitation of each submodule in the explorer π e and the advantages and robustness of π e and 3) to demonstrate that our coordinator π c can coordinate the instance pushing and grasping in structured clutter. The simulation environment and robot are kept the same with [3] for comparison purpose. We also run experiments on a real robot to show the performance of our system on "grasping the invisible" problem.…”

“…One work close to ours is visual pushing for grasping (VPG) by Zeng et al [3], which proposes a Q-learning framework to learn complementary pushing and grasping policies for robot picking task with challenging arrangements. While VPG performs target-agnostic tasks and focuses on clearing the table, our approach instead learns a critic for targetoriented manipulations and propose subtask actors to solve a more general and complex problem, "grasping the invisible".…”

“…Every 2D pixel is mapped to the 3D action execution position through the depth heightmap. Different motion angles are achieved by rotating the input heightmaps by N orientations before feeding into the networks, and there are N corresponding Q maps for pushing and grasping respectively [3]. We choose N = 16 in our system and the angle discretion is thus 22.5 • .…”

“…Coordination actor. Different from the greedy deterministic policy used in VPG [3], we propose a classifierbased coordinator, denoted as π c , to coordinate pushing and grasping. The binary classifier takes as input maximum push Q value q p = max Q p , maximum grasp Q value q g = max Q g , target border occupancy ratio…”

“…Robot pushing [1], grasping [2] or push-grasping [3] has been actively studied but mostly target-agnostic. Without incorporating target information effectively, fast adaptations (e.g., by applying the target mask) of these methods for target-oriented tasks are not successful.…”