Non-Adversarial Imitation Learning and its Connections to Adversarial Methods

Abstract

Many modern methods for imitation learning and inverse reinforcement learning, such as GAIL or AIRL, are based on an adversarial formulation. These methods apply GANs to match the expert's distribution over states and actions with the implicit state-action distribution induced by the agent's policy. However, by framing imitation learning as a saddle point problem, adversarial methods can suffer from unstable optimization, and convergence can only be shown for small policy updates. We address these problems by proposing a framework for non-adversarial imitation learning. The resulting algorithms are similar to their adversarial counterparts and, thus, provide insights for adversarial imitation learning methods. Most notably, we show that AIRL is an instance of our non-adversarial formulation, which enables us to greatly simplify its derivations and obtain stronger convergence guarantees. We also show that our non-adversarial formulation can be used to derive novel algorithms by presenting a method for offline imitation learning that is inspired by the recent ValueDice algorithm, but does not rely on small policy updates for convergence. In our simulated robot experiments, our offline method for non-adversarial imitation learning seems to perform best when using many updates for policy and discriminator at each iteration and outperforms behavioral cloning and ValueDice.

Imitation Learning: Progress, Taxonomies and Opportunities

“…Make further improvement MGAIL [7], InfoGAIL [35] Apply to other research question MAGAIL [63], GAIfO [70] Other generative model Diverse GAIL [76], GIRL [77] problem, recent research such as [3] tries to alleviate this problem by formulating the distributionmatching problem as an iterative lower-bound optimization problem.…”

Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency

“…Recently, some non-adversarial imitation learning approaches have been proposed. For example, offline nonadversarial imitation learning (Arenz and Neumann 2020) reduces the min-max in ValueDICE to policy iteration, which however still requires estimating density ratios and can potentially inherit the same issues from ValueDICE. By contrast, D2-Imitation avoids density ratio estimation and is more robust to different demonstrations.…”

Imitation by Predicting Observations

“…Unlike FORM, which uses effect models (see Figure 2) that are suitable for imitation from observations, this work models quantities that are useful primarily in conjunction with actions (modeling state-action densities and/or dynamics models for GAIL augmentation). Other recently proposed methods learn reward models either purely or partially offline (Kostrikov et al, 2020;Jarrett et al, 2020;Arenz & Neumann, 2020). This approach leans on the presence of actions in the demonstrator data.…”

“…Historically, the IRL procedure has been framed as matching the expected distribution over states and actions (or their features) along the imitator and demonstrator paths (Ng & Russell, 2000;Abbeel & Ng, 2004;Ziebart et al, 2008). As also noted in (Arenz & Neumann, 2020), we can express this as a divergence minimization problem:…”