Latent space policy search for robotics

Abstract

Abstract-Learning motor skills for robots is a hard task. In particular, a high number of degrees-of-freedom in the robot can pose serious challenges to existing reinforcement learning methods, since it leads to a highdimensional search space. However, complex robots are often intrinsically redundant systems and, therefore, can be controlled using a latent manifold of much smaller dimensionality. In this paper, we present a novel policy search method that performs efficient reinforcement learning by uncovering the low-dimensional latent space of actuator redundancies. In contrast to previous attempts at combining reinforcement learning and dimensionality reduction, our approach does not perform dimensionality reduction as a preprocessing step but naturally combines it with policy search. Our evaluations show that the new approach outperforms existing algorithms for learning motor skills with high-dimensional robots.

Dimensionality Reduction for Dynamic Movement Primitives and Application to Bimanual Manipulation of Clothes

“…While most approaches found in literature perform DR in the joint space [27], [25], [26], for comparison purposes we also derived DR in the parameter space. To do so, the procedure is equivalent to that of the previous subsections, with the exception that now the parameter r disappears and we introduce the parameter M f ≤ dN f , indicating the total number of Gaussian parameters used.…”

“…Different variants of the proposed latent space DMP representation have been tested, as well as an EM-based approach [27] adapted to the DMPs. We used episodic REPS in all the experiments and, therefore, timestep learning methods like [25] were not included in the experimentation. The application of the proposed methods does not depend on the REPS algorithm, as they can be implemented with any PS procedure using Gaussian weighted maximum likelihood estimation for reevaluating the policy parameters, such as for example PI2 [19], [20], [21].…”

“…On the contrary, if such DR is performed in the DoF space, the number of samples is larger than the DoF of the robot and, therefore, the elimination of one degree of freedom of the robot (or a linear combination of them) will not affect such unexplored space, but rather a subspace of the DoF of the robot that has a negligible impact on the outcome of the task. Other works [23], [24], [25] proposed dimensionality reduction techniques for MP representations. In our previous work [26], we showed how an iterative dimensionality reduction applied to DMPs, using policy reward evaluations to weight such DR could improve the learning of a task, and [27] used weighted maximum likelihood estimations to fit a linear projection model for MPs.…”

Dimensionality Reduction in Learning Gaussian Mixture Models of Movement Primitives for Contextualized Action Selection and Adaptation

“…In [1], the authors proposed to reduce the dimensionality of a ProMP by performing linear DR in the space of degrees of freedom (DoF) of the robot, to reduce the number of DoFs from d to r. This had the impact of reducing the dimensionality of the parameter vector ω from dN f to rN f , with r < d, N f being the number of Gaussian kernels used per DoF. While reducing the dimensionality in the robots' DoF has advantages such as a better qualitative understanding of the task, a smaller linear projection matrix which is easier to estimate, and it is also used in other approaches [18]; However, here we propose to reduce the dimensionality in the space of the Gaussian weight vectors ω. This variation is introduced to then build a GMM in the common latent parameter space and has the advantage of fine-tuning the dimensionality of the latent space, given that we can encode actions that are more different, without loosing too much information.…”

Dimensionality reduction for probabilistic movement primitives

“…Dimensionality reduction over the DoF of robots is a common approach for grasping and hand motion [6], [7]. However, it has been less used for arm robot skills.…”