Human-in-the-Loop Imitation Learning using Remote Teleoperation

Abstract

Imitation Learning is a promising paradigm for learning complex robot manipulation skills by reproducing behavior from human demonstrations. However, manipulation tasks often contain bottleneck regions that require a sequence of precise actions to make meaningful progress, such as a robot inserting a pod into a coffee machine to make coffee. Trained policies can fail in these regions because small deviations in actions can lead the policy into states not covered by the demonstrations. Intervention-based policy learning is an alternative that can address this issue -it allows human operators to monitor trained policies and take over control when they encounter failures. In this paper, we build a data collection system tailored to 6-DoF manipulation settings, that enables remote human operators to monitor and intervene on trained policies. We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions. We demonstrate that agents trained on data collected by our intervention-based system and algorithm outperform agents trained on an equivalent number of samples collected by non-interventional demonstrators, and further show that our method outperforms multiple state-ofthe-art baselines for learning from the human interventions on a challenging robot threading task and a coffee making task. Additional results and videos at https://sites.google. com/stanford.edu/iwr

Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization

“…The proxy Q value distribution shown in this section not only explains the avoidance behaviors, but also serves as a good indicator for the learned human preference. We benchmark the performance of two human-in-the-loop methods HG-DAgger (Kelly et al, 2019) and IWR (Mandlekar et al, 2020). Both methods require warming up through behavior cloning on a pre-collected dataset.…”

“…DAgger (Ross et al, 2011) and its extended methods (Kelly et al, 2019;Zhang & Cho, 2016;Hoque et al, 2021) correct the compounding error (Ross & Bagnell, 2010) of behavior cloning by periodically requesting expert to provide more demonstration. Instead of proving demonstration upon requests, Human-Gated DAgger (HG-DAgger) (Kelly et al, 2019), Expert Intervention Learning (EIL) (Spencer et al, 2020) and Intervention Weighted Regression (IWR) (Mandlekar et al, 2020) empower the expert to intervene exploration and carry the agent to safe states. However, these methods do not impose constraints to reduce human intervention and do not utilize the data from the free exploration of the agent.…”

“…Using this dataset, we evaluate passive IL method Behavior Cloning, active IL method GAIL (Ho & Ermon, 2016) and offline RL method CQL . We also run the Human-Gated DAgger (HG-DAgger) (Kelly et al, 2019) and Intervention Weighted Regression (IWR) (Mandlekar et al, 2020) as the baselines of human-in-the-loop methods based on this dataset and the human-AI copilot workflow.…”

Human-AI Shared Control via Policy Dissection

“…By incorporating human into the training, previous works successfully improve the performance on visual input control tasks such as Atari game [1,51,64]. Robotic control tasks also benefit from human feedback [28,39,58,46,65]. The other category is to have human in both training and test time to accurately accomplish humanassistive tasks.…”

Many Episode Learning in a Modular Embodied Agent via End-to-End Interaction

“…are several works showing how humans can interactively teach robotic agents, for exampleSaxena et al (2014);Paxton et al (2017);Mandlekar et al (2018);Cabi et al (2019);Mandlekar et al (2020). InSaxena et al (2014), the authors demonstrate large-scale crowd-sourcing of data for perceptual and knowledge-base components of a robotics system.…”