Rapid synchronization and accurate phase-locking of rhythmic motor primitives

Pongas, D.; Billard, Aude; Schaal, Stefan

doi:10.1109/iros.2005.1545257

Cited by 42 publications

(34 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adjustment of the primitives using sensory input can be incorporated by modifying the internal state z of the system as shown in the context of drumming (Pongas, Billard, & Schaal, 2005) and biped locomotion Schaal, Peters, Nakanishi, & Ijspeert, 2004). The equations used in order to create Eq.…”

Section: Motor Primitive Policiesmentioning

confidence: 99%

Reinforcement learning of motor skills with policy gradients

2008

View full text Add to dashboard Cite

a b s t r a c tAutonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm.

show abstract

Section: Motor Primitive Policiesmentioning

confidence: 99%

Reinforcement learning of motor skills with policy gradients

2008

View full text Add to dashboard Cite

show abstract

“…For example, the work on dynamical systems-based motor primitives [Ijspeert et al, 2002a, 2007 has allowed speeding up both imitation and reinforcement learning while, at the same time, making them more reliable. Resulting successes have shown that it is possible to rapidly learn motor primitives for complex behaviors such as tennis swings [Ijspeert et al, 2002a] with only a final target, constrained reaching [Gams and Ude, 2009], drumming [Pongas et al, 2005], biped locomotion , Nakanishi et al, 2004 and even in tasks with potential industrial application [Urbanek et al, 2004]. Although some of the presented examples, e.g., the tennis swing [Ijspeert et al, 2002a] or the T-ball batting [Peters and Schaal, 2006], are striking movements, these standard motor primitives cannot properly encode a hitting movement.…”

Section: Introductionmentioning

confidence: 99%

“…They allow acquiring new behaviors quickly and reliably both by imitation and reinforcement learning. Resulting successes have shown that it is possible to rapidly learn motor primitives for complex behaviors such as tennis-like swings [Ijspeert et al, 2002b], T-ball batting [Peters and Schaal, 2008a], drumming [Pongas et al, 2005], biped locomotion [Nakanishi et al, 2004], ball-in-a-cup [Kober and Peters, 2011a], and even in tasks with potential industrial applications [Urbanek et al, 2004]. While the examples are impressive, they do not yet address how a motor primitive can be generalized to a different behavior by trial and error without re-learning the task.…”

Section: Introductionmentioning

confidence: 99%

Learning motor skills: from algorithms to robot experiments

Kober

Peters

2014

It - Information Technology

View full text Add to dashboard Cite

Die Veröffentlichung steht unter folgender Creative Commons Lizenz: Namensnennung -Keine kommerzielle Nutzung -Keine Bearbeitung 2.0 Deutschland http://creativecommons.org/licenses/by-nc-nd/2.0/de/ Abstract Ever since the word "robot" was introduced to the English language by KarelČapek's play "Rossum's Universal Robots" in 1921, robots have been expected to become part of our daily lives. In recent years, robots such as autonomous vacuum cleaners, lawn mowers, and window cleaners, as well as a huge number of toys have been made commercially available. However, a lot of additional research is required to turn robots into versatile household helpers and companions. One of the many challenges is that robots are still very specialized and cannot easily adapt to changing environments and requirements. Since the 1960s, scientists attempt to provide robots with more autonomy, adaptability, and intelligence. Research in this field is still very active but has shifted focus from reasoning based methods towards statistical machine learning. Both navigation (i.e., moving in unknown or changing environments) and motor control (i.e., coordinating movements to perform skilled actions) are important sub-tasks.In this thesis, we will discuss approaches that allow robots to learn motor skills. We mainly consider tasks that need to take into account the dynamic behavior of the robot and its environment, where a kinematic movement plan is not sufficient. The presented tasks correspond to sports and games but the presented techniques will also be applicable to more mundane household tasks. Motor skills can often be represented by motor primitives. Such motor primitives encode elemental motions which can be generalized, sequenced, and combined to achieve more complex tasks. For example, a forehand and a backhand could be seen as two different motor primitives of playing table tennis. We show how motor primitives can be employed to learn motor skills on three different levels. First, we discuss how a single motor skill, represented by a motor primitive, can be learned using reinforcement learning. Second, we show how such learned motor primitives can be generalized to new situations. Finally, we present first steps towards using motor primitives in a hierarchical setting and how several motor primitives can be combined to achieve more complex tasks.To date, there have been a number of successful applications of learning motor primitives employing imitation learning. However, many interesting motor learning problems are high-dimensional reinforcement learning problems which are often beyond the reach of current reinforcement learning methods. We review research on reinforcement learning applied to robotics and point out key challenges and important strategies to render reinforcement learning tractable. Based on these insights, we introduce novel learning approaches both for single and generalized motor skills.For learning single motor skills, we study parametrized policy search methods and introduce a framework of reward-weighted imi...

show abstract

“…Resulting successes have shown that it is possible to rapidly learn motor primitives for complex behaviors such as tennis-like swings [1], T-ball batting [3], drumming [4], biped locomotion [5], ball-in-a-cup [6], and even in tasks with potential industrial applications [7]. The dynamical system motor primitives [1] can be adapted both spatially and temporally without changing the overall shape of the motion.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning to adjust Robot Movements to New Situations

Kober

Öztop

Peters

2010

Robotics: Science and Systems VI

155

167

View full text Add to dashboard Cite

Abstract-Many complex robot motor skills can be represented using elementary movements, and there exist efficient techniques for learning parametrized motor plans using demonstrations and self-improvement. However, in many cases, the robot currently needs to learn a new elementary movement even if a parametrized motor plan exists that covers a similar, related situation. Clearly, a method is needed that modulates the elementary movement through the meta-parameters of its representation. In this paper, we show how to learn such mappings from circumstances to meta-parameters using reinforcement learning. We introduce an appropriate reinforcement learning algorithm based on a kernelized version of the reward-weighted regression. We compare this algorithm to several previous methods on a toy example and show that it performs well in comparison to standard algorithms. Subsequently, we show two robot applications of the presented setup; i.e., the generalization of throwing movements in darts, and of hitting movements in table tennis. We show that both tasks can be learned successfully using simulated and real robots.

show abstract

Rapid synchronization and accurate phase-locking of rhythmic motor primitives

Cited by 42 publications

References 3 publications

Reinforcement learning of motor skills with policy gradients

Reinforcement learning of motor skills with policy gradients

Learning motor skills: from algorithms to robot experiments

Reinforcement Learning to adjust Robot Movements to New Situations

Contact Info

Product

Resources

About