Biped dynamic walking using reinforcement learning

Abstract

This paper presents some results from a study of biped dynamic walking using reinforcement learning. During this study a hardware biped robot was built, a new reinforcement learning algorithm as well as a new learning architecture were developed. The biped learned dynamic walking without any previous knowledge about its dynamic model. The Self Scaling Reinforcement learning algorithm was developed in order to deal with the problem of reinforcement learning in continuous action domains. The learning architecture was developed in order to solve complex control problems. It uses different modules that consist of simple controllers and small neural networks. The architecture allows for easy incorporation of new modules that represent new knowledge, or new requirements for the desired task.

Reinforcement learning of motor skills with policy gradients

“…Furthermore, in practice, already a single roll-out can suffice for an unbiased gradient estimate Spall, 2003) viable for a good policy update step, thus reducing the number of roll-outs needed. Finally, this approach has yielded the most realworld robot motor learning results (Benbrahim & Franklin, 1997;Endo et al, 2005;Gullapalli et al, 1994;Kimura & Kobayashi, 1997;Mori et al, 2004;Nakamura et al, 2004;Peters et al, 2005a). In the subsequent two sections, we will strive to explain and improve this type of gradient estimator.…”

“…Policy gradient methods are a notable exception to this statement. Starting with the pioneering work 1 of Gullapali and colleagues (Benbrahim & Franklin, 1997;Gullapalli, Franklin, & Benbrahim, 1994) in the early 1990s, these methods have been applied to a variety of robot learning problems ranging from simple control tasks (e.g., balancing a ball on a beam (Benbrahim, Doleac, Franklin, & Selfridge, 1992), and pole balancing (Kimura & Kobayashi, 1998)) to complex learning tasks involving many degrees of freedom such as learning of complex motor skills (Gullapalli et al, 1994;Mitsunaga, Smith, Kanda, Ishiguro, & Hagita, 2005;Miyamoto et al, 1995Miyamoto et al, , 1996Peters & Schaal, 2006;Peters, Vijayakumar, & Schaal, 2005a) and locomotion (Endo, Morimoto, Matsubara, Nakanishi, & Cheng, 2005;Kimura & Kobayashi, 1997;Kohl & Stone, 2004;Mori, Nakamura, aki Sato, & Ishii, 2004;Sato, Nakamura, & Ishii, 2002;Tedrake, Zhang, & Seung, 2005).…”

Policy Gradient Methods for Robotics

“…Furthermore, in practice, already a single roll-out can suffice for an unbiased gradient estimate [21], [25] viable for a good policy update step, thus reducing the amount of roll-outs needed. Finally, this approach has yielded the most real-world robotics results [1], …”

“…Policy gradient methods are a notable exception to this statement. Starting with the pioneering work of Gullapali, Franklin and Benbrahim [1], [2] in the early 1990s, these methods have been applied to a variety of robot learning problems ranging from simple control tasks (e.g., balancing a ball-on a beam [3], and pole-balancing [4]) to complex learning tasks involving many degrees of freedom such as learning of complex motor skills [2], [5], [6] and locomotion [7]- [14] 1 . The advantages of policy gradient methods for robotics are numerous.…”

“…The advantages of policy gradient methods for robotics are numerous. Among the most important ones are that the policy representations can be chosen so that it is meaningful for the task and can incorporate previous domain knowledge, that often fewer parameters are needed in the learning process than in value-function based approaches and that there is a variety of different algorithms for policy gradient estimation in the 1 Recent advances in robot learning [15] clearly indicate that policy gradient methods are currently the most feasible choice for robotics and that few other algorithms appear applicable. Some methods used in robotics estimate and apply policy gradients but are not well-known as such, e.g., differential dynamic programming usually estimates a model-based gradient and the PEGASUS [16] trick is often used to estimate finite difference gradients efficiently.…”