An introduction to stochastic control theory, path integrals and reinforcement learning

Kappen, Hilbert J.

doi:10.1063/1.2709596

Cited by 96 publications

(117 citation statements)

References 25 publications

(26 reference statements)

Supporting

Mentioning

116

Contrasting

Order By: Relevance

“…However, one could argue that a sampling procedure to compute the path integral corresponds to a learning of the environment. A discussion on this line of thought can be found in (Kappen, 2007).…”

Section: Discussionmentioning

confidence: 99%

Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

Broek

Wiegerinck

Kappen

2008

jair

View full text Add to dashboard Cite

In this article we consider the issue of optimal control in collaborative multi-agent systems with stochastic dynamics. The agents have a joint task in which they have to reach a number of target states. The dynamics of the agents contains additive control and additive noise, and the autonomous part factorizes over the agents. Full observation of the global state is assumed. The goal is to minimize the accumulated joint cost, which consists of integrated instantaneous costs and a joint end cost. The joint end cost expresses the joint task of the agents. The instantaneous costs are quadratic in the control and factorize over the agents. The optimal control is given as a weighted linear combination of single-agent to single-target controls. The single-agent to single-target controls are expressed in terms of diffusion processes. These controls, when not closed form expressions, are formulated in terms of path integrals, which are calculated approximately by Metropolis-Hastings sampling. The weights in the control are interpreted as marginals of a joint distribution over agent to target assignments. The structure of the latter is represented by a graphical model, and the marginals are obtained by graphical model inference. Exact inference of the graphical model will break down in large systems, and so approximate inference methods are needed. We use naive mean field approximation and belief propagation to approximate the optimal control in systems with linear dynamics. We compare the approximate inference methods with the exact solution, and we show that they can accurately compute the optimal control. Finally, we demonstrate the control method in multi-agent systems with nonlinear dynamics consisting of up to 80 agents that have to reach an equal number of target states.

show abstract

Section: Discussionmentioning

confidence: 99%

Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

Broek

Wiegerinck

Kappen

2008

jair

View full text Add to dashboard Cite

show abstract

“…In Kappen [7], control problems are associated with animal behavior. Living organisms, including human beings, employ cognitive resources to take decisions.…”

Section: Some Specific Applications and Contributionsmentioning

confidence: 99%

“…The fundamental point, though, is that one is dealing with the future; the plan is set now, at t=0, for an horizon that starts at t=0 and extends to a pre-defined future date. Therefore, as emphasized by Kappen [7], the control problem is stochastic in nature. There is uncertainty associated with future outcomes and the best the agent can do is to compute the optimal trajectory of some control variable(s) contingent on how the system is supposed to evolve.…”

Section: Why Stochastic Controls?mentioning

confidence: 99%

On the Relevance of Stochastic Controls

Gomes¹

2014

J Appl Computat Math

View full text Add to dashboard Cite

OverviewMany scientifically relevant problems share two salient features. First, they are dynamic; second, they involve uncertainty. Hence, it is natural that researchers would be concerned with the study of events taking place in stochastic dynamic settings. In these settings, the decision-maker will typically want to select optimal paths for an array of control variables in order to maximize or minimize the current value of a sequence of future expected outcomes. In this article, we defend the argument that exploring techniques and applications in the field of stochastic optimal control theory is vital for the advancement of applied science. Although solid steps have been taken, in the last few years, to consolidate the theory of stochastic controls and to make this an adequate tool to address many important problems in multiple fields of knowledge, further work is still necessary to accomplish new insights and to unveil new results in an area of extreme complexity, where the search for efficient paths is many times hampered by the high degree of underlying uncertainty. The benchmark optimization problemStochastic optimal control problems are concerned with the intertemporal optimization (maximization or minimization) of an objective function subject to one or more constraints that, in continuous time, acquire the form of stochastic differential equations. The objective function typically respects to the expected value of a sequence of utility levels that range from the initial date t=0 to some future horizon. In the case of economic problems, the horizon is commonly assumed as infinite and the future is discounted at a constant rate ρ>0. Taking the autonomous case, for which time is not an explicit argument of the problem's functional, the objective function acquires the following shape, is assumed to be a real-valued continuous and differentiable function. Two categories of variables constitute the arguments of f, namely the state variables, ( ) n x t ∈  and the control variables, ( ) m u t ∈  . State variables are those that have their laws of motion determined by the differential equations respecting to the problem's constraints; control variables are the ones that the decision-maker is able to control in order to pursue the specified dynamic goal.The constraints underlying the optimization problem are, as mentioned above, stochastic differential equations. A generic specification is the following, ( ) If, for all 0≤s

show abstract

“…Optimal control [1], [2] aims at finding optimal behavior given a cost function and the dynamics of the system. Typically, the cost function consists of several objectives that need to be traded off by the experimenter.…”

Section: Introductionmentioning

confidence: 99%

Optimal control and inverse optimal control by distribution matching

Arenz

Abdulsamad

Neumann

2016

2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Abstract-Optimal control is a powerful approach to achieve optimal behavior. However, it typically requires a manual specification of a cost function which often contains several objectives, such as reaching goal positions at different time steps or energy efficiency. Manually trading-off these objectives is often difficult and requires a high engineering effort. In this paper, we present a new approach to specify optimal behavior. We directly specify the desired behavior by a distribution over future states or features of the states. For example, the experimenter could choose to reach certain mean positions with given accuracy/variance at specified time steps. Our approach also unifies optimal control and inverse optimal control in one framework. Given a desired state distribution, we estimate a cost function such that the optimal controller matches the desired distribution. If the desired distribution is estimated from expert demonstrations, our approach performs inverse optimal control. We evaluate our approach on several optimal and inverse optimal control tasks on non-linear systems using incremental linearizations similar to differential dynamic programming approaches.

show abstract

An introduction to stochastic control theory, path integrals and reinforcement learning

Cited by 96 publications

References 25 publications

Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

On the Relevance of Stochastic Controls

Optimal control and inverse optimal control by distribution matching

Contact Info

Product

Resources

About