Optimal control as a graphical model inference problem

Kappen, Hilbert J.; Gómez, Vicenç; Opper, Manfred

doi:10.1007/s10994-012-5278-7

Cited by 267 publications

(284 citation statements)

References 27 publications

(44 reference statements)

Supporting

Mentioning

279

Contrasting

Order By: Relevance

“…The equivalence in computational complexity is reflected in the fact that many procedures are found in both approximate solutions to optimal control and Bayesian inference. Examples here include minimisation of Kullback-Leibler divergences (Todorov 2008;Kappen et al 2009), and expectation maximisation (Toussaint and Storkey 2006), both of which can be formulated as minimising variational free energy (Neal and Hinton 1998). The main contribution of this paper concerns the interpretation of optimality, as opposed to an algorithmic contribution.…”

Section: Discussionmentioning

confidence: 99%

Active inference and agency: optimal control without cost functions

2012

View full text Add to dashboard Cite

This paper describes a variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty. We show that optimal control can be cast as active inference. In active inference, both action and posterior beliefs about hidden states minimise a free energy bound on the negative log-likelihood of observed states, under a generative model. In this setting, reward or cost functions are absorbed into prior beliefs about state transitions and terminal states. Effectively, this converts optimal control into a pure inference problem, enabling the application of standard Bayesian filtering techniques. We then consider optimal trajectories that rest on posterior beliefs about hidden states in the future. Crucially, this entails modelling control as a hidden state that endows the generative model with a representation of agency. This leads to a distinction between models with and without inference on hidden control states; namely, agency-free and agency-based models, respectively.

show abstract

Section: Discussionmentioning

confidence: 99%

Active inference and agency: optimal control without cost functions

2012

View full text Add to dashboard Cite

show abstract

“…In this study, we use an information-theoretic model of bounded rational decision-making Braun, 2012, 2013;Braun and Ortega, 2014; that has precursors in the economic literature (McKelvey and Palfrey, 1995;Mattsson and Weibull, 2002;Sims, 2003Sims, , 2005Sims, , 2006Sims, , 2010Wolpert, 2006) and that is closely related to recent advances in the information theory of perception-action systems (Todorov, 2007(Todorov, , 2009Still, 2009;Friston, 2010;Peters et al, 2010;Tishby and Polani, 2011;Daniel et al, 2012Daniel et al, , 2013Kappen et al, 2012;Rawlik et al, 2012;Rubin et al, 2012;Neymotin et al, 2013;Tkačik and Bialek, 2014;Palmer et al, 2015). The basis of this approach is formalized by a free energy principle that trades off expected utility, and the cost of computation that is required to adapt the system accordingly in order to achieve high utility.…”

Section: Introductionmentioning

confidence: 99%

Bounded Rationality, Abstraction, and Hierarchical Decision-Making: An Information-Theoretic Optimality Principle

et al. 2015

View full text Add to dashboard Cite

Abstraction and hierarchical information processing are hallmarks of human and animal intelligence underlying the unrivaled flexibility of behavior in biological systems. Achieving such flexibility in artificial systems is challenging, even with more and more computational power. Here, we investigate the hypothesis that abstraction and hierarchical information processing might in fact be the consequence of limitations in information-processing power. In particular, we study an information-theoretic framework of bounded rational decision-making that trades off utility maximization against information-processing costs. We apply the basic principle of this framework to perception-action systems with multiple information-processing nodes and derive bounded-optimal solutions. We show how the formation of abstractions and decision-making hierarchies depends on information-processing costs. We illustrate the theoretical ideas with example simulations and conclude by formalizing a mathematically unifying optimization principle that could potentially be extended to more complex systems.

show abstract

“…As a result, the optimal control can be estimated using Monte Carlo sampling. See [21,22,45,47] for earlier reviews and references.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Importance Sampling for Control and Inference

Kappen

Ruiz

2016

J Stat Phys

View full text Add to dashboard Cite

Path integral (PI) control problems are a restricted class of non-linear control problems that can be solved formally as a Feynman-Kac PI and can be estimated using Monte Carlo sampling. In this contribution we review PI control theory in the finite horizon case. We subsequently focus on the problem how to compute and represent control solutions. We review the most commonly used methods in robotics and control. Within the PI theory, the question of how to compute becomes the question of importance sampling. Efficient importance samplers are state feedback controllers and the use of these requires an efficient representation. Learning and representing effective state-feedback controllers for non-linear stochastic control problems is a very challenging, and largely unsolved, problem. We show how to learn and represent such controllers using ideas from the cross entropy method. We derive a gradient descent method that allows to learn feed-back controllers using an arbitrary parametrisation. We refer to this method as the path integral cross entropy method or PICE. We illustrate this method for some simple examples. The PI control methods can be used to estimate the posterior distribution in latent state models. In neuroscience these problems arise when estimating connectivity from neural recording data using EM. We demonstrate the PI control method as an accurate alternative to particle filtering.

show abstract

Optimal control as a graphical model inference problem

Cited by 267 publications

References 27 publications

Active inference and agency: optimal control without cost functions

Active inference and agency: optimal control without cost functions

Bounded Rationality, Abstraction, and Hierarchical Decision-Making: An Information-Theoretic Optimality Principle

Adaptive Importance Sampling for Control and Inference

Contact Info

Product

Resources

About