An Algorithmic Perspective on Imitation Learning

Abstract

As robots and other intelligent agents move from simple environments and problems to more complex, unstructured settings, manually programming their behavior has become increasingly challenging and expensive. Often, it is easier for a teacher to demonstrate a desired behavior rather than attempt to manually engineer it. This process of learning from demonstrations, and the study of algorithms to do so, is called imitation learning. This work provides an introduction to imitation learning. It covers the underlying assumptions, approaches, and how they relate; the rich set of algorithms developed to tackle the problem; and advice on effective tools and implementation. We intend this paper to serve two audiences. First, we want to familiarize machine learning experts with the challenges of imitation learning, particularly those arising in robotics, and the interesting theoretical and practical distinctions between it and more familiar frameworks like statistical supervised learning theory and reinforcement learning. Second, we want to give roboticists and experts in applied artificial intelligence a broader appreciation for the frameworks and tools available for imitation learning. We organize our work by dividing imitation learning into directly replicating desired behavior (sometimes called behavioral cloning [Bain and Sammut, 1996]) and learning the hidden objectives of the desired behavior from demonstrations (called inverse optimal control [Kalman, 1964] or inverse reinforcement learning [Russell, 1998]). In addition to method analysis, we discuss the design decisions a practitioner must make when selecting an imitation learning approach. Moreover, application examples-such as robots that play table tennis [Kober and Peters, 2009] and programs that play the game of Go [Silver et al., 2016]-illustrate the properties and motivations behind different forms of imitation learning. We conclude by presenting a set of open questions and point towards possible future research directions.

SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards

“…This baseline directly learns a policy from an initial set of demonstrations using supervised learning. This approach is called behavioral cloning (see the survey of imitation learning given by Osa et al (2018)); in each of our experiments, we describe the policy models used. It is important to note that this approach requires fully observed demonstrations.…”

“…This fully observed setting is typical of the literature on imitation learning (Osa et al, 2018), where one uses a function approximation to learn the state to action mapping from these samples. However, one may have a more limited access to a limited supervisor, where only the states are observed.…”

Recurrent Neural Network Control of a Hybrid Dynamical Transfemoral Prosthesis with EdgeDRNN Accelerator

“…where i is the index of a sequence in a batch, B the batch size, y i t = τ d,i pk,t , τ d,i pa,t the vector of labels, i t the loss of time step t, L the loss of a batch with sequences of length T . The L1 loss is used instead of the L2 loss because of its robustness to outliers [33].…”

Modeling Task Uncertainty for Safe Meta-Imitation Learning

“…Some papers mention the significance of evaluating predictive uncertainty to ensure the safety of the controller in the context of imitation learning. Previous papers have pointed out that the main problems in imitation learning lies in the inherent ambiguity of demonstrations (Goo and Niekum, 2019 ) or the discrepancy between training and test conditions that can lead robots to perform unexpected actions (Pomerleau, 1989 ; Osa et al, 2018 ). In practice, one possible solution is measuring the predictive uncertainty, and if the robots are uncertain about their prediction, they can stop performing actions and request that experts provide additional demonstrations (Thakur et al, 2019 ).…”