SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards

Abstract

Learning to imitate expert behavior from demonstrations can be challenging, especially in environments with high-dimensional, continuous observations and unknown dynamics. Supervised learning methods based on behavioral cloning (BC) suffer from distribution shift: because the agent greedily imitates demonstrated actions, it can drift away from demonstrated states due to error accumulation. Recent methods based on reinforcement learning (RL), such as inverse RL and generative adversarial imitation learning (GAIL), overcome this issue by training an RL agent to match the demonstrations over a long horizon. Since the true reward function for the task is unknown, these methods learn a reward function from the demonstrations, often using complex and brittle approximation techniques that involve adversarial training. We propose a simple alternative that still uses RL, but does not require learning a reward function. The key idea is to provide the agent with an incentive to match the demonstrations over a long horizon, by encouraging it to return to demonstrated states upon encountering new, out-of-distribution states. We accomplish this by giving the agent a constant reward of r = +1 for matching the demonstrated action in a demonstrated state, and a constant reward of r = 0 for all other behavior. Our method, which we call soft Q imitation learning (SQIL), can be implemented with a handful of minor modifications to any standard Q-learning or off-policy actor-critic algorithm. Theoretically, we show that SQIL can be interpreted as a regularized variant of BC that uses a sparsity prior to encourage long-horizon imitation. Empirically, we show that SQIL outperforms BC and achieves competitive results compared to GAIL, on a variety of image-based and low-dimensional tasks in Box2D, Atari, and MuJoCo. This paper is a proof of concept that illustrates how a simple imitation method based on RL with constant rewards can be as effective as more complex methods that use learned rewards.Preprint. Under review.

JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

“…We conduct experiments on MineRL environment [Guss et al, 2019]. Our approach is built based on existing RL algorithms including SQIL [Reddy et al, 2019], PPO [Schulman et al, 2017], DQfD [Hester et al, 2018]. We present the details of our approach as well as the experiment settings in Appendix 5.…”

“…Baselines. We compare our approach with several baselines in two categories: 1) end-to-end learning methods used in prior work and in the official implementation provided, including BC [Kanervisto et al, 2020], SQIL [Reddy et al, 2019], Rainbow [Hessel et al, 2018], DQfD [Hester et al, 2018], PDDDQN [Schaul et al, 2015;Van Hasselt et al, 2016;Wang et al, 2016]; all these baselines are trained within 8 million samples with default hyper-parameters.…”

“…Models. We describe the models that we use in our hierarchical framework -meta-policy: we use neural networks with 3-layer MLPs and ReLU activation; stage 1 (Treechop): we use SQIL [Reddy et al, 2019], which learns from demonstrations with a constant reward function, and we also plug our A2RL technique into SQIL to boost perception; stage 2 (WoodAxe): we use our proposed EBC that learns policies by purely imitating from human demonstrations; stage 3 (Stone): we use DQfD [Hester et al, 2018] but only use it to pre-train our sub-policy with demonstrations and do not interact with the environment; stage 4 (Furnace): we use random search method that randomly samples action in the action space; stage 5 (Iron Ore): we use PPO [Schulman et al, 2017] with our proposed A2RL and DSIL; stage 6 (IronAxe): we use random search method as in stage 4; stage 7 (Diamond): we use the same training procedure as used in stage 5. Note that the action space in each stage is automatically extracted from the structured demonstrations.…”

Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency

“…The discriminator is crucial for D2-Imitation to work properly and guarantees convergence to expert performance. We perform ablations on the discriminator and compare the training performance of D2-Imitation with one variant: one without any discriminator (denoted without-discriminator), which just puts on-policy samples to B 0 , assigns 0 reward to them and gives +1 reward to demonstration samples in B + , an idea adopted in Soft-Q Imitation Learning (SQIL) (Reddy, Dragan, and Levine 2019). Figure 5 inator is applied.…”

Supervised Advantage Actor-Critic for Recommender Systems

“…We call this Supervised Negative Q-learning (SNQN). Another interpretation of negative sampling in RL is imitation learning under sparse reward settings [23]. Different from SQN, which only performs RL on positive actions (clicks, views, etc.…”