Zero-Shot Visual Imitation

Abstract

The current dominant paradigm for imitation learning relies on strong supervision of expert actions to learn both what and how to imitate. We pursue an alternative paradigm wherein an agent first explores the world without any expert supervision and then distills its experience into a goal-conditioned skill policy with a novel forward consistency loss. In our framework, the role of the expert is only to communicate the goals (i.e., what to imitate) during inference. The learned policy is then employed to mimic the expert (i.e., how to imitate) after seeing just a sequence of images demonstrating the desired task. Our method is "zero-shot" in the sense that the agent never has access to expert actions during training or for the task demonstration at inference. We evaluate our zero-shot imitator in two real-world settings: complex rope manipulation with a Baxter robot and navigation in previously unseen office environments with a TurtleBot. Through further experiments in VizDoom simulation, we provide evidence that better mechanisms for exploration lead to learning a more capable policy which in turn improves end task performance. Videos, models, and more details are available at https://pathak22.github.io/zeroshot-imitation/. * Denotes equal contribution.

Generative Adversarial Imitation from Observation

“…One way to approach the aforementioned problem is to "learn to imitate" (as opposed to imitation learning), i.e., by doing some preprocessing, enable the agent to follow a single demonstration exactly. Two such approaches are proposed by Nair et al (2017) and Pathak et al (2018). These methods first learn an inverse dynamics model through selfsupervised exploration, and then use it to infer the demonstrator's action at each step and perform that in the environment.…”

Unsupervised Control Through Non-Parametric Discriminative Rewards

“…We are, however, interested in environments with continuous highdimensional observation spaces. While there is extensive prior work on learning goal-conditioned policies (Kaelbling, 1993;Schaul et al, 2015;Andrychowicz et al, 2017;Held et al, 2017;Pathak et al, 2018), the reward function is often hand-crafted, limiting generality of the approaches. In the few cases where the reward is learned, the learning objective is typically tied to a pre-specified notion of visual similarity.…”

“…Experiments showed that once a UPN is trained the state representation it learned can be used to construct a reward function for visually specified goals. Bridging goal-conditioned policy learning and imitation learning, Pathak et al (2018) learns a goal-conditioned policy and a dynamics model with supervised learning without expert trajectories, and present zero-shot imitation of trajectories from a sequence of images of a desired task.…”

Imitating Latent Policies from Observation

“…A recent approach aimed to learn from observations by first learning how to imitate in a self-supervised manner, then given a task, attempt it zero-shot (Pathak et al, 2018). However, this approach requires learning in the agent's environment first rather than initially learning from the observations.…”

“…A recent approach for overcoming these issues is to learn an initial self-supervised model for how to imitate by collecting experiences within the environment and then using this learned model to infer policies from expert observations (Pathak et al, 2018;Torabi et al, 2018a). However, unguided exploration can be risky in many real-world sce-narios and costly to obtain.…”