Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization

Abstract

Recent progress in state-only imitation learning extends the scope of applicability of imitation learning to real-world settings by relieving the need for observing expert actions. However, existing solutions only learn to extract a state-to-action mapping policy from the data, without considering how the expert plans to the target. This hinders the ability to leverage demonstrations and limits the flexibility of the policy. In this paper, we introduce Decoupled Policy Optimization (DePO), which explicitly decouples the policy as a high-level state planner and an inverse dynamics model. With embedded decoupled policy gradient and generative adversarial training, DePO enables knowledge transfer to different action spaces or state transition dynamics, and can generalize the planner to out-of-demonstration state regions. Our in-depth experimental analysis shows the effectiveness of DePO on learning a generalized target state planner while achieving the best imitation performance. We demonstrate the appealing usage of DePO for transferring across different tasks by pre-training, and the potential for cotraining agents with various skills.

D2EA: Depict the Epidemic Picture of COVID-19

“…Each trajectory τ e is composed of a sequence of observations τ e = {o e t } T t=1 . A prevalent approach to address the ILfO problem involves transforming it into a Reinforcement Learning (RL) problem by defining proxy rewards based on the agent's trajectory τ and the expert demonstrations: {r t } T −1 t=1 := f r (τ, D e ), where f r represents a criterion for reward assignment (Torabi et al, 2018b;Yang et al, 2019;Lee et al, 2021;Jaegle et al, 2021;Liu et al, 2022;Huang et al, 2023). Subsequently, RL is employed to maximize the expected discounted sum of rewards:…”

“…Some approaches train an inverse dynamics model by the agent's collected data, and use this model to infer the expert's missing action information (Nair et al, 2017;Torabi et al, 2018a;Pathak et al, 2018;Radosavovic et al, 2021). Recent work also integrates the inverse dynamics model with proxy-reward-based algorithms (Liu et al, 2022;Ramos et al, 2023). Taking a different approach, Edwards et al (2019) learns a forward dynamics model on a latent action space.…”

“…Canonical imitation algorithms, such as behavior cloning (Bain & Sammut, 1995;Ross et al, 2011;Daftry et al, 2017), can not be directly applied to ILfO, as they rely on access to the expert's actions for behavior recovery. To deal with ILfO problems, one prominent category of approaches involves getting proxy rewards based on the distribution of the agent's and the expert's visited states (Torabi et al, 2018b;Yang et al, 2019;Lee et al, 2021;Kidambi et al, 2021;Jaegle et al, 2021;Liu et al, 2022). These approaches first derive stepwise reward signals through techniques like occupancy measure matching (Ho & Ermon, 2016) or trajectory matching (Haldar et al, 2023a), and then employ reinforcement learning (RL) to optimize the expected cumulative reward.…”