Adaptive Importance Sampling for Control and Inference

Abstract

Path integral (PI) control problems are a restricted class of non-linear control problems that can be solved formally as a Feynman-Kac PI and can be estimated using Monte Carlo sampling. In this contribution we review PI control theory in the finite horizon case. We subsequently focus on the problem how to compute and represent control solutions. We review the most commonly used methods in robotics and control. Within the PI theory, the question of how to compute becomes the question of importance sampling. Efficient importance samplers are state feedback controllers and the use of these requires an efficient representation. Learning and representing effective state-feedback controllers for non-linear stochastic control problems is a very challenging, and largely unsolved, problem. We show how to learn and represent such controllers using ideas from the cross entropy method. We derive a gradient descent method that allows to learn feed-back controllers using an arbitrary parametrisation. We refer to this method as the path integral cross entropy method or PICE. We illustrate this method for some simple examples. The PI control methods can be used to estimate the posterior distribution in latent state models. In neuroscience these problems arise when estimating connectivity from neural recording data using EM. We demonstrate the PI control method as an accurate alternative to particle filtering.

Geometric approach to optimal nonequilibrium control: Minimizing dissipation in nanomagnetic spin systems

“…[24] uses path sampling techniques to harvest nonequilibrium protocols in proportion to their average dissipation. Trajectory space Monte Carlo techniques have also been developed for use in stochastic optimal control theory to iteratively refine importance sampling distributions [25,26,32], exploiting the connection between importance sampling and optimal control [33,34]. With a bias that favors low dissipation, Gingrich et al [24] explore an ensemble of low dissipation protocols and show that there is a large number of protocols with a dissipation near the minimum achievable value.…”

Topology-guided path integral approach for stochastic optimal control in cluttered environment

“…Note that all the estimations are unbiased [7,6]. Especially, it is proven that the variance of estimation decreases as u in becomes closer to the real optimal control u * [6].…”

“…In [6], path-integral formula is utilized to construct a state-dependent feedback controller and theoretical analysis on how sampling strategies affect the estimation results is presented. In [7], the cross entropy method was applied to build an efficient importance sampler that reduces estimation variance. In [8], the rapidly-exploring random tree (RRT) algorithm was used to help the importance sampler to pick valuable samples.…”

Variational Inference for Stochastic Differential Equations

“…It turns out, that the gradient with respect to θ can be estimated iteratively by a Monte Carlo method using adaptive importance sampling. [28,29]…”