Path integrals and symmetry breaking for optimal control theory

Abstract

This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton-Jacobi-Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton-Jacobi equation to the Schrödinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process, that can be computed by stochastic integration or by the evaluation of a path integral. It is shown, how in the deterministic limit the PMP formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as MC sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise.

Optimal control of molecular dynamics using Markov state models

“…A brief version is provided for the reader's convenience; for the technical details we refer to, e.g., [36]. Let L u = " + (u rV (x)) · r denote the infinitesimal generator of (6). Here the superscript indicates the explicit dependence on the control variable.…”

“…Here the superscript indicates the explicit dependence on the control variable. We have to show that the solutions to (10) yield optimal controls that maximize (8)-(9) subject to (6). Now choose a…”

“…Available methods for the numerical solution of high dimensional HJB equations include Markov chain approximations [2], monotone schemes [3,4], or methods designed for relatively specific problems [5,6]. Nonetheless we are not aware of a single article on optimal control of molecular dynamics (MD) using dynamic programming principles, although the interest in controlling molecular dynamics simulations has already started more than a decade ago with the development of Targeted MD or Steered MD [7,8], and laser control of (open) quantum systems [9]; see [10] for a recent survey of approaches from molecular physics, chemistry, and optical control of quantum molecular dynamics.…”

Solving the Hamilton-Jacobi-Bellman equation for a stochastic system with state constraints

“…Additionally, we require both W and R to be positive definite and bounded everywhere on Ω, but otherwise impose no restrictions on them. Contrary to the assumptions in previous work [9,10,14] and the work of Kappen [7] and Broek et al [12] they are no longer required to relate to the inverse of eachother. As formulated, the control u and the noise w enters the state equation via the same matrix G. However, the problem can easily be reformulated such that the control and noise enter via different matrices as long as they have the same column space [14].…”

Learning motor skills: from algorithms to robot experiments

“…In recent years, general reinforcement learning has yielded three kinds of policy search approaches that have translated particularly well into the domain of robotics: (i) policy gradients approaches based on likelihood-ratio estimation [Sutton et al, 1999], (ii) policy updates inspired by expectation-maximization [Toussaint et al, 2010], and (iii) the path integral methods [Kappen, 2005]. Likelihood-ratio policy gradient methods rely on perturbing the motor command instead of comparing in policy space.…”