Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection

Levine, Sergey; Pastor, Peter; Krizhevsky, Alex; Quillen, Deirdre

doi:10.48550/arxiv.1603.02199

Cited by 42 publications

(80 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In bin-picking, objects are located in a pile and are often small and light. Methods that target bin-picking [1]- [3], [8], [9] benefit from the pile structure as objects have more stable equilibriuma and as such collisions with obstacles does not jeopardise the pile structure. In comparison, methods that target grasping in structured clutter [4]-where objects are mostly larger, heavier, and packed together-need to avoid all collisions as unintended contact between the robot and objects can easily tip objects over and thus change the scene structure.…”

Section: B Grasping In Cluttermentioning

confidence: 99%

“…In cluttered scenes, the most successful methods have resorted to the simper 4-DOF planar top-down grasps [1]- [3], [8] instead of full 6-DOF spatial grasps [4], [7], [22]. While 4-DOF grasps are simpler, the restriction on the arm motion they imply hinders the robot's ability to grasp specific objects [4], [7], especially with structured clutter.…”

Section: B Grasping In Cluttermentioning

confidence: 99%

“…With target-driven grasp sampling methods [4] it is possible to specify the object to grasp opposed to target-free methods [1]- [3], [7]- [9] which oftentimes simply chooses the object it can attain the best grasp on. Our method is target-driven as it segments objects and plan object-specific grasps using these segments.…”

Section: B Grasping In Cluttermentioning

confidence: 99%

“…The stable resting poses were calculated using trimesh https:// github.com/mikedh/trimesh3 We will release the datasets after acceptance of the manuscript.…”

mentioning

confidence: 99%

See 3 more Smart Citations

DDGC: Generative Deep Dexterous Grasping in Clutter

Lundell¹,

Verdoja²,

Kyrki³

2021

Preprint

View full text Add to dashboard Cite

Recent advances in multi-fingered robotic grasping have enabled fast 6-Degrees-Of-Freedom (DOF) single object grasping. Multi-finger grasping in cluttered scenes, on the other hand, remains mostly unexplored due to the added difficulty of reasoning over obstacles which greatly increases the computational time to generate high-quality collision-free grasps. In this work we address such limitations by introducing DDGC, a fast generative multi-finger grasp sampling method that can generate high quality grasps in cluttered scenes from a single RGB-D image. DDGC is built as a network that encodes scene information to produce coarse-to-fine collision-free grasp poses and configurations. We experimentally benchmark DDGC against the simulated-annealing planner in GraspIt! on 1200 simulated cluttered scenes and 7 real world scenes. The results show that DDGC outperforms the baseline on synthesizing highquality grasps and removing clutter while being 5 times faster. This, in turn, opens up the door for using multi-finger grasps in practical applications which has so far been limited due to the excessive computation time needed by other methods 1 .

show abstract

Section: B Grasping In Cluttermentioning

confidence: 99%

Section: B Grasping In Cluttermentioning

confidence: 99%

Section: B Grasping In Cluttermentioning

confidence: 99%

“…The stable resting poses were calculated using trimesh https:// github.com/mikedh/trimesh3 We will release the datasets after acceptance of the manuscript.…”

mentioning

confidence: 99%

See 2 more Smart Citations

DDGC: Generative Deep Dexterous Grasping in Clutter

Lundell¹,

Verdoja²,

Kyrki³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…End-to-end learning techniques, such as deep reinforcement learning algorithms, combine both processes to directly build policies [3], and can theoretically be used to learn new tasks and adapt to environmental variations in such open-ended scenarios. However, they require very large learning datasets, which is not suited to real robotics tasks where evaluating the outcome of actions or policies is slow and costly [4], except in the rare cases when many real robots can be used in parallel [3]. They are also notably slow to converge when reward states are sparse, which is often the case in tasks necessitating to reach some very specific goal states using only partial, high-dimensional and noisy obervations of the world.…”

Section: Introductionmentioning

confidence: 99%

From exploration to control: learning object manipulation skills through novelty search and local adaptation

Kim,

Coninx,

Doncieux

2019

Preprint

View full text Add to dashboard Cite

Programming a robot to deal with open-ended tasks remains a challenge, in particular if the robot has to manipulate objects. Launching, grasping, pushing or any other object interaction can be simulated but the corresponding models are not reversible and the robot behavior thus cannot be directly deduced. These behaviors are hard to learn without a demonstration as the search space is large and the reward sparse. We propose a method to autonomously generate a diverse repertoire of simple object interaction behaviors in simulation. Our goal is to bootstrap a robot learning and development process with limited informations about what the robot has to achieve and how. This repertoire can be exploited to solve different tasks in reality thanks to a proposed adaptation method or could be used as a training set for data-hungry algorithms.The proposed approach relies on the definition of a goal space and generates a repertoire of trajectories to reach attainable goals, thus allowing the robot to control this goal space. The repertoire is built with an off-the-shelf simulation thanks to a quality diversity algorithm. The result is a set of solutions tested in simulation only. It may result in two different problems: (1) as the repertoire is discrete and finite, it may not contain the trajectory to deal with a given situation or (2) some trajectories may lead to a behavior in reality that differs from simulation because of a reality gap. We propose an approach to deal with both issues by using a local linearization between the motion parameters and the observed effects. Furthermore, we present an approach to update the existing solution repertoire with the tests done on the real robot. The approach has been validated on two different experiments on the Baxter robot: a ball launching and a joystick manipulation tasks.

show abstract

Dealing with Ambiguity in Robotic Grasping via Multiple Predictions

Ghazaei

Laina

Rupprecht

et al. 2019

Computer Vision – ACCV 2018

View full text Add to dashboard Cite

Humans excel in grasping and manipulating objects because of their life-long experience and knowledge about the 3D shape and weight distribution of objects. However, the lack of such intuition in robots makes robotic grasping an exceptionally challenging task. There are often several equally viable options of grasping an object. However, this ambiguity is not modeled in conventional systems that estimate a single, optimal grasp position. We propose to tackle this problem by simultaneously estimating multiple grasp poses from a single RGB image of the target object. Further, we reformulate the problem of robotic grasping by replacing conventional grasp rectangles with grasp belief maps, which hold more precise location information than a rectangle and account for the uncertainty inherent to the task. We augment a fully convolutional neural network with a multiple hypothesis prediction model that predicts a set of grasp hypotheses in under 60 ms, which is critical for real-time robotic applications. The grasp detection accuracy reaches over 90% for unseen objects, outperforming the current state of the art on this task.

show abstract

Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection

Cited by 42 publications

References 0 publications

DDGC: Generative Deep Dexterous Grasping in Clutter

DDGC: Generative Deep Dexterous Grasping in Clutter

From exploration to control: learning object manipulation skills through novelty search and local adaptation

Dealing with Ambiguity in Robotic Grasping via Multiple Predictions

Contact Info

Product

Resources

About