Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

Broek, Bart van den; Wiegerinck, Wim; Kappen, Bert

doi:10.1613/jair.2473

Cited by 75 publications

(51 citation statements)

References 41 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, the role of noise in symmetry breaking phenomena was investigated in the context of stochastic optimal control. In [22], the path integral formalism is extended for stochastic optimal control of multi-agent systems, which is not unlike our multi DOF control systems.…”

Section: Related Workmentioning

confidence: 99%

Variable Impedance Control - A Reinforcement Learning Approach

Buchli¹,

Theodorou²,

Stulp³

et al. 2010

Robotics: Science and Systems VI

View full text Add to dashboard Cite

Abstract-One of the hallmarks of the performance, versatility, and robustness of biological motor control is the ability to adapt the impedance of the overall biomechanical system to different task requirements and stochastic disturbances. A transfer of this principle to robotics is desirable, for instance to enable robots to work robustly and safely in everyday human environments. It is, however, not trivial to derive variable impedance controllers for practical high DOF robotic tasks. In this contribution, we accomplish such gain scheduling with a reinforcement learning approach algorithm, PI 2 (Policy Improvement with Path Integrals). PI 2 is a model-free, sampling based learning method derived from first principles of optimal control. The PI 2 algorithm requires no tuning of algorithmic parameters besides the exploration noise. The designer can thus fully focus on cost function design to specify the task. From the viewpoint of robotics, a particular useful property of PI 2 is that it can scale to problems of many DOFs, so that RL on real robotic systems becomes feasible. We sketch the PI 2 algorithm and its theoretical properties, and how it is applied to gain scheduling. We evaluate our approach by presenting results on two different simulated robotic systems, a 3-DOF Phantom Premium Robot and a 6-DOF Kuka Lightweight Robot. We investigate tasks where the optimal strategy requires both tuning of the impedance of the end-effector, and tuning of a reference trajectory. The results show that we can use path integral based RL not only for planning but also to derive variable gain feedback controllers in realistic scenarios. Thus, the power of variable impedance control is made available to a wide variety of robotic systems and practical applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Variable Impedance Control - A Reinforcement Learning Approach

Buchli¹,

Theodorou²,

Stulp³

et al. 2010

Robotics: Science and Systems VI

View full text Add to dashboard Cite

show abstract

“…In this paper, we present a probabilistic reinforcement learning approach, which is derived from the framework of stochastic optimal control and path integrals, based on the original work of [14]. As will be detailed in the sections below, this approach makes an appealing theoretical connection between value function approximation using the stochastic HJB equations and direct policy learning by approximating a path integral, i.e., by solving a statistical inference problem from sample roll-outs.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement learning of full-body humanoid motor skills

Stulp

Buchli

Theodorou

et al. 2010

2010 10th IEEE-RAS International Conference on Humanoid Robots

View full text Add to dashboard Cite

Applying reinforcement learning to humanoid robots is challenging because humanoids have a large number of degrees of freedom and state and action spaces are continuous. Thus, most reinforcement learning algorithms would become computationally infeasible and require a prohibitive amount of trials to explore such high-dimensional spaces. In this paper, we present a probabilistic reinforcement learning approach, which is derived from the framework of stochastic optimal control and path integrals. The algorithm, called Policy Improvement with Path Integrals (PI 2 ), has a surprisingly simple form, has no open tuning parameters besides the exploration noise, is modelfree, and performs numerically robustly in high dimensional learning problems. We demonstrate how PI 2 is able to learn fullbody motor skills on a 34-DOF humanoid robot. To demonstrate the generality of our approach, we also apply PI 2 in the context of variable impedance control, where both planned trajectories and gain schedules for each joint are optimized simultaneously. This paper is accompanied by a movie:

show abstract

“…It is not hard to show [25] that the optimal control (17) may be related to the expected value of the first discretized increment x 1 of the trajectory x N as 1…”

Section: Path Integral Constructionmentioning

confidence: 99%

“…Related works incorporating the PI framework for multiagent systems designed control for systems in which agents cooperatively compute their control from a marginalization of the joint probability distribution of the group's system trajectory [25,26]. However, to the best of our knowledge, this paper contains the first use of the PI framework for a robotic team without explicit inter-agent communication.…”

Section: Introductionmentioning

confidence: 99%

Distributed path integral feedback control based on Kalman smoothing for unicycle formations

Anderson

Milutinović

2013

2013 American Control Conference

View full text Add to dashboard Cite

We consider the problem of a team of nonholonomic agents that must independently compute a discounted, infinite-horizon optimal feedback control from a Hamilton-Jacobi-Bellman equation in order to drive the team into a distance-based formation, without explicit communication. In this work, the uncertainty in a neighboring agent's control input is modeled by Brownian motion, which allows the solution to the Hamilton-Jacobi-Bellman equation to be written as a path integral over the agents' future trajectories. We show how this representation allows the formation control problem to be transformed into independent Kalman smoothing problems, avoiding the computational issues that are typically associated with computing an optimal feedback control for highdimensional systems. A numerical example in which five agents form a regular pentagon is provided.

show abstract

Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

Cited by 75 publications

References 41 publications

Variable Impedance Control - A Reinforcement Learning Approach

Variable Impedance Control - A Reinforcement Learning Approach

Reinforcement learning of full-body humanoid motor skills

Distributed path integral feedback control based on Kalman smoothing for unicycle formations

Contact Info

Product

Resources

About