AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning

Abstract

Most approaches in reinforcement learning (RL) are data-hungry and specific to fixed environments. In this paper, we propose a principled framework for adaptive RL, called AdaRL, that adapts reliably to changes across domains. Specifically, we construct a generative environment model for the structural relationships among variables in the system and embed the changes in a compact way, which provides a clear and interpretable picture for locating what and where the changes are and how to adapt. Based on the environment model, we characterize a minimal set of representations, including both domain-specific factors and domain-shared state representations, that suffice for reliable and low-cost transfer. Moreover, we show that by explicitly leveraging a compact representation to encode changes, we can adapt the policy with only a few samples without further policy optimization in the target domain. We illustrate the efficacy of AdaRL through a series of experiments that allow for changes in different components of Cartpole and Atari games.Preprint. Under review.

Learning Latent Causal Dynamics

“…We evaluate LiLY on the modified cartpole (Huang et al, 2021) video dataset and compare the performances with LEAP. Modified Cartpole is a nonlinear dynamical system with cart positions x t and pole angles θ t as the true state variables.…”

“…Modified Cartpole The Cartpole problem (Huang et al, 2021) "consists of a cart and a vertical pendulum attached to the cart using a passive pivot joint. The cart can move left or right.…”

“…The original dataset (Huang et al, 2021) introduces "two change factors respectively for the state transition dynamics θ dyn k : varying gravity and varying mass of the cart, and a change factor in the observation function θ obs k that is the image noise level. Fig.…”

“…Left Right Table S1. The settings of source and target domains for modified Cartpole experiments (Huang et al, 2021).…”

A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

“…This can be leveraged to learn dynamics models that explicitly ignore irrelevant factors in prediction or to compute improved sample complexity bounds for policy learning [42], and seems particularly relevant for generalisation as additional structure in the context set could map onto the factored structure in the transition and reward functions. An initial example of using a similar formalism to a factored MDP in a multi-domain RL setting is [43], although it does not target the zero-shot policy transfer setting directly. We hope to see more work applying these kinds of structural assumptions to generalisation problems.…”