Reinforcement learning of full-body humanoid motor skills

Abstract

Applying reinforcement learning to humanoid robots is challenging because humanoids have a large number of degrees of freedom and state and action spaces are continuous. Thus, most reinforcement learning algorithms would become computationally infeasible and require a prohibitive amount of trials to explore such high-dimensional spaces. In this paper, we present a probabilistic reinforcement learning approach, which is derived from the framework of stochastic optimal control and path integrals. The algorithm, called Policy Improvement with Path Integrals (PI 2 ), has a surprisingly simple form, has no open tuning parameters besides the exploration noise, is modelfree, and performs numerically robustly in high dimensional learning problems. We demonstrate how PI 2 is able to learn fullbody motor skills on a 34-DOF humanoid robot. To demonstrate the generality of our approach, we also apply PI 2 in the context of variable impedance control, where both planned trajectories and gain schedules for each joint are optimized simultaneously. This paper is accompanied by a movie:

Variance modulated task prioritization in Whole-Body Control

“…Through three emblematic scenarios, we showed how variable task weights resolve a broad set of issues encountered in multi-task execution with minimal tuning and in a reactive manner. In addition to the variance to weights mapping, we developed a method of computing variance for a single trajectory demonstration using a covariance function (8). This tool is essential in cases where only one trajectory has been provided for the task, as in trajectory generation.…”

“…, can be obtained through multiple demonstrations 2 as in [5], [7], [8], [9], or computed from scratch. The concatenation of these position means and variances, respectively yields M Υ and V Υ for the given trajectory, Υ.…”

“…Task variance is measured from the variability of the learned motions and may be adapted based on new demonstrations [7], [8], [9]. An inverse relationship between the task variance and the K p gain is then formed to regulate the attractor term during the movement.…”

Learning of closed-loop motion control

“…[3], [4]). Among different approaches in this area, reinforcement learning algorithms have shown good performance both in simulation and in real world applications.…”

“…The goal is to control the robot from an initial state to a final state within 6 seconds. At the final time, the center of the ball should be at the [3,1] T meter from its starting position and the robot should also have zero velocity, zero tilt angles and tilt angle rates as well as zero change in the heading. The designed cost function has the same form as in (3) where intermediate l (x, u) and final h (x) costs are defined as follows:…”