Model-based contextual policy search for data-efficient generalization of robot skills

Abstract

In robotics, lower-level controllers are typically used to make the robot solve a specific task in a fixed context. For example, the lower-level controller can encode a hitting movement while the context defines the target coordinates to hit. However, in many learning problems the context may change between task executions. To adapt the policy to a new context, we utilize a hierarchical approach by learning an upper-level policy that generalizes the lower-level controllers to new contexts. A common approach to learn such upper-level policies is to use policy search. However, the majority of current contextual policy search approaches are model-free and require a high number of interactions with the robot and its environment. Model-based approaches are known to significantly reduce the amount of robot experiments, however, current model-based techniques cannot be applied straightforwardly to the problem of learning contextual upper-level policies. They rely on specific parametrizations of the policy and the reward function, which are often unrealistic in the contextual policy search formulation. In this paper, we propose a novel model-based contextual policy search algorithm that is able to generalize lower-level controllers, and is data-efficient. Our approach is based on learned probabilistic forward models and information theoretic policy search. Unlike current algorithms, our method does not require any assumption on the parametrization of the policy or the reward function. We show on complex simulated robotic tasks and in a real robot experiment that the proposed learning framework speeds up the learning process by up to two orders of magnitude in comparison to existing methods, while learning high quality policies.

Low-Level Control of a Quadrotor With Deep Model-Based Reinforcement Learning

“…Functionality of MBRL is evident in simulation for multiple tasks in low data regimes, including quadrupeds [20] and manipulation tasks [21]. Low-level MBRL control (i.e., with direct motor input signals) of an RC car has been demonstrated experimentally, but the system is of lower dimensionality and has static stability [22].…”

Factored Contextual Policy Search with Bayesian optimization

“…While they map experience gained in one context to another, they do so for estimating discrete outcome probabilities and not for improving the policy. GP-REPS [19] iteratively learns a transition model of the system using a Gaussian process (GP) [20], which is then used to generate trajectories offline for updating the policy. The authors consider generating additional samples for artificial contexts, but they do not define an explicit factorization.…”

Experiments with Hierarchical Reinforcement Learning of Multiple Grasping Policies

“…The policy update using contextual REPS is formulated as a constraint optimization problem, max π µs(s) π(θ|s)R(θ, s)dθds (12) s.t. ≥ µs(s)KL (π(θ|s)||q(θ|s)) ds, 1 = π(θ|s)ds (13) For details, please refer to the original study and its extensions [15,16]. Contextual REPS models the policy as a Gaussian policy…”

“…Grasping motions for different grasp types are learned as independent policies π k l by the contextual relative entropy policy search (REPS) algorithm [15][16][17]. The learned policy π k l generates the motion parameter θ using the local features of the estimated grasping part s.…”