Sample-Efficient Robot Motion Learning using Gaussian Process Latent Variable Models

Abstract

Robotic manipulators are reaching a state where we could see them in household environments in the following decade. Nevertheless, such robots need to be easy to instruct by lay people. This is why kinesthetic teaching has become very popular in recent years, in which the robot is taught a motion that is encoded as a parametric function -usually a Movement Primitive (MP)-. This approach produces trajectories that are usually suboptimal, and the robot needs to be able to improve them through trial-and-error. Such optimization is often done with Policy Search (PS) reinforcement learning, using a given reward function. PS algorithms can be classified as model-free, where neither the environment nor the reward function are modelled, or model-based, which can use a surrogate model of the reward function and/or a model for the dynamics of the task.However, MPs can become very high-dimensional in terms of parameters, which constitute the search space, so their optimization often requires too many samples. In this paper, we assume we have a robot motion task characterized with an MP of which we cannot model the dynamics. We build a surrogate model for the reward function, that maps an MP parameter latent space (obtained through a Mutual-information-weighted Gaussian Process Latent Variable Model) into a reward. While we do not model the task dynamics, using mutual information to shrink the task space makes it more consistent with the reward and so the policy improvement is faster in terms of sample efficiency.

A topological extension of movement primitives for curvature modulation and sampling of robot motion

“…Additionally, it can be combined with vision algorithms to find relevant geometries in the environment, as done by Pumarola et al 2017 with an application to SLAM. Finally, the methodology can be applied with dimensionality reduction techniques, such as Gaussian Process Latent Variable Models (GPLVM) (Li and Chen 2016) in order to obtain a reduced-dimension feature space for policy learning, as in the works of Koganti et al 2017, Koganti et al 2019and Delgado-Guerrero et al 2020…”

Contextual Policy Search for Micro-Data Robot Motion Learning through Covariate Gaussian Process Latent Variable Models

“…As a future work, we intend to perform updates of c-GPLVM during the learning process, to consider task modulation according to time-varying context data, to create artificial data when needed by means of random variations of the context vector, to complete reward information with user ratings, and to use GPLVM extensions such as Bayesian GPLVM [32], [33]. Furthermore, this work, together with the one presented in [14], is planned to be extended and improved, including a more exhaustive evaluation of the algorithm, comparing it with several of the aforementioned state-of-the-art methods.…”

“…In this paper, we assume that tasks at hand can be modelled as MPs, physically executed and evaluated by means of a reward function considered as a black box, and prior information on parameters is available through initial demonstrations, but instead, we cannot model their dynamics. On this basis, the proposed solution continues along the research line presented in [14]. Therefore, to speed up convergence, we build a surrogate model of reward and apply Bayesian Optimization (BO) in the latent space arisen from a Dimensionality Reduction (DR) of the PS parameter space, since BO algorithms do not perform well with high-dimensional search spaces.…”

“…Therefore, to speed up convergence, we build a surrogate model of reward and apply Bayesian Optimization (BO) in the latent space arisen from a Dimensionality Reduction (DR) of the PS parameter space, since BO algorithms do not perform well with high-dimensional search spaces. This enhanced method is implemented in a convenient manner to ensure contextual adaptation, and it significantly increments the DR power and data efficiency with respect to [14]. In particular, Upper Confidence Bound (UCB) [15], [16] and Gaussian Process (GP) regression [17] have been used to learn the model of the expected return, and covariate Gaussian Process Latent Variable Models (c-GPVLM) [18] have been applied for context-aware DR.…”

Task-Adaptive Robot Learning From Demonstration With Gaussian Process Models Under Replication

“…(GMM) [4], Kernelized Movement Primitives (KMP) [5] and Gaussian Process models (GP) [6]. In a recent work [7], we presented a GP-based LfD framework, which we adopt as a basis for this paper.…”