Reinforcement learning of motor skills with policy gradients

Abstract

a b s t r a c tAutonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm.

Preference-Based Policy Learning

“…A main third approach aims at direct policy learning, using policy gradient methods [20] or global optimization methods [28]. Direct policy learning most usually assumes a parametric policy space Θ; policy learning aims at finding the optimal θ * parameter in the sense of a policy return function J:…”

“…The first case is when J is analytically known on a continuous policy space Θ ⊂ IR D . It then comes naturally to use a gradient-based optimization approach, gradually moving the current policy θ t along the gradient ∇J [20] (θ t+1 = θ t + α t ∇ θ J(θ t )). The main issues concern the adjustment of α t , the possible use of the inverse Hessian, and the rate of convergence toward a (local) optimum.…”

“…The generation of new policies relies on black-box optimization; expected-global improvement methods [6] and gradient methods [20] are not applicable since the policy return estimate is not defined on the parametric representation space Θ. New policies are thus generated using a stochastic optimization method, more precisely a (1 + λ)-ES algorithm [2].…”

“…Many machine learning approaches in robotics, ranging from direct policy learning [3] and learning by imitation [7] to reinforcement learning [20] and inverse optimal control [1,14], rely on the use of simulators used as full motion and world model of the robot. Naturally, the actual performance of the learned policies depends on the accuracy and calibration of the simulator (see [22]).…”

“…Compared to learning by demonstration [7], the demonstrated trajectories are feasible by construction (whereas the teacher and the robot usually have different degrees of freedom). Compared to policy gradient approaches [20], the continued interaction with the expert offers some means to detect and avoid the convergence to local optima, through the expert's feedback.…”