Robust and Efficient Transfer Learning with Hidden Parameter Markov Decision Processes

Abstract

An intriguing application of transfer learning emerges when tasks arise with similar, but not identical, dynamics. Hidden Parameter Markov Decision Processes (HiP-MDP) embed these tasks into a low-dimensional space; given the embedding parameters one can identify the MDP for a particular task. However, the original formulation of HiP-MDP had a critical flaw: the embedding uncertainty was modeled independently of the agent's state uncertainty, requiring an arduous training procedure. In this work, we apply a Gaussian Process latent variable model to jointly model the dynamics and the embedding, leading to a more elegant formulation, one that allows for better uncertainty quantification and thus more robust transfer.

Constraint Sampling Reinforcement Learning: Incorporating Expertise for Faster Learning

“…We created a simulator for multiple heterogeneous patient groups by perturbing the internal hidden parameters of the system following Killian, Konidaris, and Doshi-Velez (2017). We learn an optimal decision policy for 3 different groups, and then use the known optimal policies as constraints to learn in a new group which may or may not be similar to a group with a known policy.…”

Multi-policy Grounding and Ensemble Policy Learning for Transfer Learning with Dynamics Mismatch

“…Option learning [Yang et al, 2020b], policy function aggregation [Barekatain et al, 2019], policy shaping [Plisnier et al, 2019], and bayesian inference based mixture density network [Gimelfarb et al, 2020] are different methods proposed to fully utilize multiple source policies. Another stream of research assumes the existence of hidden parameters that define transition dynamics of environment [Killian et al, 2017;Perez et al, 2020;Yang et al, 2020a]. These studies exploit the advantage of hidden parameters that simplify the estimation of dynamics differences, hence facilitate transfer learning.…”

Visual Transfer For Reinforcement Learning Via Wasserstein Domain Confusion

“…Some work directly addresses dynamics transfer where the transition function changes, but states, actions, and rewards are identical. Some work (Killian et al 2017;Doshi-Velez and Konidaris 2016;Yao et al 2018) parameterizes the transition function of an MDP and learning a conditional policy that can transfer between such tasks. Other work generates a curriculum to learn generalizable policies that can adapt to MDPs with differing dynamics (Mysore, Platt, and Saenko 2019).…”