Incremental bootstrapping of parameterized motor skills

Abstract

Many motor skills have an intrinsic, low-dimensional parameterization, e.g. reaching through a grid to different targets. Repeated policy search for new parameterizations of such a skill is inefficient, because the structure of the skill variability is not exploited. This issue has been previously addressed by learning mappings from task parameters to policy parameters. In this work, we introduce a bootstrapping technique that establishes such parameterized skills incrementally. The approach combines iterative learning with state-of-the-art black-box policy optimization. We investigate the benefits of incrementally learning parameterized skills for efficient policy retrieval and show that the number of required rollouts can be significantly reduced when optimizing policies for novel tasks. The approach is demonstrated for several parameterized motor tasks including upper-body reaching motion generation for the humanoid robot COMAN

Bootstrapping of Parameterized Skills Through Hybrid Optimization in Task and Policy Spaces

“…This work extends our previous method (Queißer et al, 2016) and its contribution aims at the experimental verification of the following hypotheses:…”

“…The representation of the parameterized skill in relation to the number (gray scale) of consolidated samples is visualized. © 2016 IEEE, reproduced with permission from (Queißer et al, 2016).…”

“…Figure 2 | Bootstrapping loop of parameterized skills as proposed in (Queißer et al, 2016). (a) System overview including simulation of a 10-DOF planar arm, parameterized reaching target at T 2 , parameterized skill and optimization module.…”

“…In a similar way as (Baranes et al, 2013), we generalize for new task parameterizations to transfer optimization results. We investigate an incremental algorithm to establish parameterized skills that reuse previous experience to successively improve the initialization of the optimization process (Queißer et al, 2016). Thereby we are able to incorporate state-of-the-art optimization of the policy, i.e., by CMA-ES, instead of optimizing meta-parameters of policies (Kober et al, 2012) and do not rely on library based approaches (Mülling et al, 2010).…”

“…Based on previous experiments (Queißer et al, 2016), we argue for a utilization of the parameterized skill as a projection of the low dimensional manifold of task-space to the high dimensional search space of policy parameters. By performing a policy optimization in this low dimensional manifold, a further speed-up, in terms of number of rollouts, during the optimization process can be observed.…”