Scaling data-driven robotics with reward sketching and batch reinforcement learning

Cabi, Serkan; Colmenarejo, Sergio Gómez; Novikov, Alexander; Konyushova, Ksenia; Reed, Scott; Jeong, Rae; Żołna, Konrad; Aytar, Yusuf; Budden, David; Vecerik, Mel; Sushkov, Oleg; Broneske, David; Scholz, Jonathan; Denil, Misha; Freitas, Nando de; Wang, Ziyu

doi:10.15607/rss.2020.xvi.076

Cited by 43 publications

(48 citation statements)

References 50 publications

(60 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3D environments, requiring a much more sophisticated dynamics model. Third, we use a richer form of feedback, reward sketches (Cabi et al, 2020), rather than sparse label-based feedback.…”

Section: Learned Dynamics Models In Prior Workmentioning

confidence: 99%

Safe Deep RL in 3D Environments using Human Feedback

Matthew¹,

Varma²,

Kumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Agents should avoid unsafe behaviour during both training and deployment. This typically requires a simulator and a procedural specification of unsafe behaviour. Unfortunately, a simulator is not always available, and procedurally specifying constraints can be difficult or impossible for many real-world tasks. A recently introduced technique, ReQueST, aims to solve this problem by learning a neural simulator of the environment from safe human trajectories, then using the learned simulator to efficiently learn a reward model from human feedback. However, it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans -whether sufficient pixel-based neural simulator quality can be achieved, and whether the human data requirements are viable in terms of both quantity and quality. In this paper we answer this question in the affirmative, using ReQueST to train an agent to perform a 3D first-person object collection task using data entirely from human contractors. We show that the resulting agent exhibits an order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning.

show abstract

“…3D environments, requiring a much more sophisticated dynamics model. Third, we use a richer form of feedback, reward sketches (Cabi et al, 2020), rather than sparse label-based feedback.…”

Section: Learned Dynamics Models In Prior Workmentioning

confidence: 99%

Safe Deep RL in 3D Environments using Human Feedback

Matthew¹,

Varma²,

Kumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Much like our work, a number of prior works have studied how learning from broad datasets can enhance generalization in robot learning [16,33,56,13,22,24,10,5]. These works DVD is trained to predict if two videos are completing the same task or not.…”

Section: Robotic Learning From Large Datasetsmentioning

confidence: 99%

“…have largely studied the problem of collecting large and diverse robotic datasets in scalable ways [28,22,10,53,7] as well as techniques for learning general purpose policies from this style of data in an offline [13,5] or online [33,29,24] fashion. While our motivation of achieving generalization by learning from diverse data heavily overlaps with the above works, our approach fundamentally differs in that it aims to sidestep the challenges associated with collecting diverse robotic data by instead leveraging existing human data sources.…”

Section: Robotic Learning From Large Datasetsmentioning

confidence: 99%

Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos

Chen¹,

Nair²,

Finn³

2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot's ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language, collecting high quality datasets of robotic interaction at scale remains an open challenge. In contrast, "in-the-wild" videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. In this work, we propose a simple approach, Domain-agnostic Video Discriminator (DVD), that learns multitask reward functions by training a discriminator to classify whether two videos are performing the same task, and can generalize by virtue of learning from a small amount of robot data with a broad dataset of human videos. We find that by leveraging diverse human datasets, this reward function (a) can generalize zero shot to unseen environments, (b) generalize zero shot to unseen tasks, and (c) can be combined with visual model predictive control to solve robotic manipulation tasks on a real WidowX200 robot in an unseen environment from a single human demo.

show abstract

“…As discussed previously, our goal is to train control policies in the real world in a timescale that is feasible for industrial use cases. [20] demonstrated that it is possible to train a USB-insertion policy directly from pixels, but it took 8 Calculate relative pose T t rel = T t i × T −1 i 7:…”

Section: Pre-trained Visual Featuresmentioning

confidence: 99%

Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study

Luo

Sushkov

Pevceviciute

et al. 2021

Robotics: Science and Systems XVII

Self Cite

View full text Add to dashboard Cite

Over the past several years there has been a considerable research investment into learning-based approaches for tasks inspired by industrial manufacturing, but despite significant progress, these techniques have yet to be adopted by in the realworld. We argue that it is the prohibitively large design space for Deep Reinforcement Learning (DRL), rather than algorithmic limitations per se, that are truly responsible for this lack of adoption. Pushing these techniques into the industrial mainstream requires a paradigm which differs significantly from the academic mindset. In this paper we define criteria for industryoriented DRL, and perform a thorough comparison according to these criteria of one family of learning approaches, DRL from demonstration, against results of a professional industrial integrator on the recently established NIST assembly benchmark. We explain the design choices, representing several years of investigation, which enabled our DRL system to consistently outperform the integrator's baseline in terms of both speed and reliability. Finally, we conclude with a competition between our DRL system and a human on a challenge task of insertion into a randomly moving target. This study suggests that DRL is capable of outperforming not only established engineered approaches, but the human motor system as well, and that there remains significant room for improvement. Videos can be found on our project website:https://sites.google.com/view/shield-nist.

show abstract

Scaling data-driven robotics with reward sketching and batch reinforcement learning

Cited by 43 publications

References 50 publications

Safe Deep RL in 3D Environments using Human Feedback

Safe Deep RL in 3D Environments using Human Feedback

Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos

Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study

Contact Info

Product

Resources

About