Factored Contextual Policy Search with Bayesian optimization

Abstract

Scarce data is a major challenge to scaling robot learning to truly complex tasks, as we need to generalize locally learned policies over different task contexts. Contextual policy search offers data-efficient learning and generalization by explicitly conditioning the policy on a parametric context space. In this paper, we further structure the contextual policy representation. We propose to factor contexts into two components: target contexts that describe the task objectives, e.g. target position for throwing a ball; and environment contexts that characterize the environment, e.g. initial position or mass of the ball. Our key observation is that experience can be directly generalized over target contexts. We show that this can be easily exploited in contextual policy search algorithms. In particular, we apply factorization to a Bayesian optimization approach to contextual policy search both in sampling-based and active learning settings. Our simulation results show faster learning and better generalization in various robotic domains. See our supplementary video: https://youtu.be/MNTbBAOufDY.

Reinforcement Learning in Sparse-Reward Environments With Hindsight Policy Gradients

“…In this context, hindsight refers to the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended. Prior to our work, hindsight has been limited to off-policy reinforcement learning algorithms that rely on experience replay (Andrychowicz et al, 2017) and policy search based on Bayesian optimization (Karkus et al, 2016;Pinsler et al, 2019).…”

“…In earlier work, Karkus, Kupcsik, Hsu, and Lee (2016) introduced hindsight to policy search based on Bayesian optimization (Metzen, Fabisch, & Hansen, 2015). This work was recently extended by Pinsler, Karkus, Kupcsik, Hsu, and Lee (2019).…”

“…Suppose that the reward r(s, g) is known for every combination of state s and goal g, as in previous work on hindsight (Andrychowicz et al, 2017;Karkus et al, 2016;Pinsler et al, 2019). In that case, it is possible to evaluate a trajectory obtained while trying to achieve an original goal g for an alternative goal g. This information can be exploited using a central result based on importance sampling.…”

Contextual Policy Search for Micro-Data Robot Motion Learning through Covariate Gaussian Process Latent Variable Models

“…Subsequently, the robot skills can be honed by updating the MP parameters through trial-and-error within the framework of Policy Search (PS), a branch of Reinforcement Learning (RL) [2] responsible for resolving which trajectories to evaluate in consideration of the rewards of each execution. Thus, PS algorithms have proved successful in several robotic applications [3], including the contextual case, in which robots are required to adapt to changing environments [4], [5], [6], [7].…”