Compliant skills acquisition and multi-optima policy search with EM-based reinforcement learning

Abstract

The democratization of robotics technology and the development of new actuators progressively bring robots closer to humans. The applications that can now be envisaged drastically contrast with the requirements of industrial robots. In standard manufacturing settings, the criterions used to assess performance are usually related to the robot's accuracy, repeatability, speed or stiffness. Learning a control policy to actuate such robots is characterized by the search of a single solution for the task, with a representation of the policy consisting of moving the robot through a set of points to follow a trajectory. With new environments such as homes and offices populated with humans, the reproduction performance is portrayed differently. These robots are expected to acquire rich motor skills that can be generalized to new situations, while behaving safely in the vicinity of users. Skills acquisition can no longer be guided by a single form of learning, and must instead combine different approaches to continuously create, adapt and refine policies. The family of search strategies based on expectation-maximization (EM) looks particularly promising to cope with these new requirements. The exploration can be performed directly in the policy parameters space, by refining the policy together with exploration parameters represented in the form of covariances. With this formulation, RL can be extended to a multi-optima search problem in which several policy alternatives can be considered. We present here two applications exploiting EM-based exploration strategies, by considering parameterized policies based on dynamical systems, and by using Gaussian mixture models for the search of multiple policy alternatives.

Learning motions from demonstrations and rewards with time-invariant dynamical systems based policies

“…It is applied to learning the "ball in a cup" game, using a DMP as a parameterized policy. A similar algorithm was used in [5] in a pancake flipping task using a different parameterized policy consisting of a time-dependent mixture of proportional-derivative systems. PI 2 is a policy search method derived from the first principles of stochastic optimal control [14].…”

“…They have convenient properties, such as the possibility to allow non-constrained learning while ensuring qualitative behavior such as global stability at an attractor point or convergence to a limit cycle. There have been numerous successful demonstrations of learning from demonstration and reinforcement learning in such motion representations [18,19,5] learning motions for a large variety of tasks.…”

“…These two paradigms are complementary and often used in conjunction, e.g. by using LfD as a means to find a good initial task model which is subsequently refined by RL [5,6].…”

Simultaneous on-line Discovery and Improvement of Robotic Skill options

“…This number is initially chosen to be larger than the number of possible optima, and redundant options are deleted as the algorithm runs, with care taken to avoid premature deletion. Calinon et al [15] uses a Gaussian Mixture Model to estimate the structure of the cost function J, mapping the cost as a probability density function that depends on the policy parameters. Exploration noise is decayed manually, rather than automating it with covariance matrix adaptation.…”

“…Exploration noise is decayed manually, rather than automating it with covariance matrix adaptation. The key difference between the algorithms in [7], [15] and SODIRS is that the former consider multiple skill options that are able to solve the same task, whereas SODIRS learns multiple options for multiple task variations, without requiring the user to specify the task parameter space.…”