Learning Hand-Eye Coordination for Robotic Grasping with Large-Scale Data Collection

Levine, Sergey; Pástor, Peter; Krizhevsky, Alex; Quillen, Deirdre

doi:10.1007/978-3-319-50115-4_16

Cited by 210 publications

(153 citation statements)

References 12 publications

Supporting

Mentioning

147

Contrasting

Unclassified

Order By: Relevance

“…Behavior cloning [31,37,34,23] is a form of supervised learning that can learn sensorimotor policies from off-line collected data. The only requirements are pairs of input sensory observations associated with expert actions.…”

Section: Conditional Imitation Learningmentioning

confidence: 99%

Exploring the Limitations of Behavior Cloning for Autonomous Driving

Codevilla

Santana

López³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

318

315

View full text Add to dashboard Cite

Figure 1. Driving scenarios from our new benchmark where the agent needs to react to dynamic changes in the environment, handle clutter (only part of the environment is causally relevant), and predict complex sensorimotor controls (lateral and longitudinal). We show that Behavior Cloning yields state-of-the-art policies in these complex scenarios and investigate its limitations. AbstractDriving requires reacting to a wide variety of complex environment conditions and agent behaviors. Explicitly modeling each possible scenario is unrealistic. In contrast, imitation learning can, in theory, leverage data from large fleets of human-driven cars. Behavior cloning in particular has been successfully used to learn simple visuomotor policies end-to-end, but scaling to the full spectrum of driving behaviors remains an unsolved problem. In this paper, we propose a new benchmark to experimentally investigate the scalability and limitations of behavior cloning. We show that behavior cloning leads to state-of-the-art results, including in unseen environments, executing complex lateral and longitudinal maneuvers without these reactions being explicitly programmed. However, we confirm well-known limitations (due to dataset bias and overfitting), new generalization issues (due to dynamic objects and the lack of a causal model), and training instability requiring further research before behavior cloning can graduate to real-world driving. We will release our benchmark and code.

show abstract

Section: Conditional Imitation Learningmentioning

confidence: 99%

Exploring the Limitations of Behavior Cloning for Autonomous Driving

Codevilla

Santana

López³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

318

315

View full text Add to dashboard Cite

show abstract

“…The strong nonlinear nature of the system makes it difficult to build a vehicle model without significant uncertainty, limiting the effectiveness of modelbased solutions. However, neural networks have shown great potential for optimising nonlinear, high-dimensional control systems [40], [41], [101]- [106]. For instance, reinforcement learning can learn an optimal control policy through interaction with the environment, without knowledge of the system model [50].…”

Section: B Longitudinal Control Systemsmentioning

confidence: 99%

A Survey of Deep Learning Applications to Autonomous Vehicle Control

Kuutti

Bowden

Jin

et al. 2021

IEEE Trans. Intell. Transport. Syst.

384

148

View full text Add to dashboard Cite

Designing a controller for autonomous vehicles capable of providing adequate performance in all driving scenarios is challenging due to the highly complex environment and inability to test the system in the wide variety of scenarios which it may encounter after deployment. However, deep learning methods have shown great promise in not only providing excellent performance for complex and non-linear control problems, but also in generalising previously learned rules to new scenarios. For these reasons, the use of deep learning for vehicle control is becoming increasingly popular. Although important advancements have been achieved in this field, these works have not been fully summarised. This paper surveys a wide range of research works reported in the literature which aim to control a vehicle through deep learning methods. Although there exists overlap between control and perception, the focus of this paper is on vehicle control, rather than the wider perception problem which includes tasks such as semantic segmentation and object detection. The paper identifies the strengths and limitations of available deep learning methods through comparative analysis and discusses the research challenges in terms of computation, architecture selection, goal specification, generalisation, verification and validation, as well as safety. Overall, this survey brings timely and topical information to a rapidly evolving field relevant to intelligent transportation systems.

show abstract

“…Like several others, we apply RL techniques to the problem of robotic manipulation (see abovementioned [10], [13], [15], [18], [21] and survey [22]). RL is appealing for robotic control for several reasons.…”

Section: B Reinforcement Learningmentioning

confidence: 99%

Learning Manipulation Skills via Hierarchical Spatial Attention

Gualtieri

Platt

2020

IEEE Trans. Robot.

View full text Add to dashboard Cite

Learning generalizable skills in robotic manipulation has long been challenging due to realworld sized observation and action spaces. One method for addressing this problem is attention focus -the robot learns where to attend its sensors and irrelevant details are ignored. However, these methods have largely not caught on due to the difficulty of learning a good attention policy and the added partial observability induced by a narrowed window of focus. This article addresses the first issue by constraining gazes to a spatial hierarchy. For the second issue, we identify a case where the partial observability induced by attention does not prevent Q-learning from finding an optimal policy. We conclude with real-robot experiments on challenging pick-place tasks demonstrating the applicability of the approach.

show abstract

Learning Hand-Eye Coordination for Robotic Grasping with Large-Scale Data Collection

Cited by 210 publications

References 12 publications

Exploring the Limitations of Behavior Cloning for Autonomous Driving

Exploring the Limitations of Behavior Cloning for Autonomous Driving

A Survey of Deep Learning Applications to Autonomous Vehicle Control

Learning Manipulation Skills via Hierarchical Spatial Attention

Contact Info

Product

Resources

About