Probabilistic inference for solving discrete and continuous state Markov Decision Processes

Toussaint, Marc; Storkey, Amos

doi:10.1145/1143844.1143963

Cited by 190 publications

(177 citation statements)

References 5 publications

(3 reference statements)

Supporting

Mentioning

174

Contrasting

Unclassified

Order By: Relevance

“…A first step in this direction was already made in Wiegerinck et al (2006), van den Broek et al (2008a). In this case, we have considered the KL-stag-hunt game and shown that BP provides a good approximation and allows to analyze the behavior of large systems, where exact inference is not feasible.…”

Section: Discussionmentioning

confidence: 86%

“…The KL control approach proposed in this paper also bears some relation to the EM approach of Toussaint and Storkey (2006), who consider the discounted reward case with 0, 1 rewards. The posterior can be considered a mixture over times at which rewards are incorporated.…”

Section: Related Workmentioning

confidence: 99%

“…This class of control problem has been applied to multi-agent problems using a graphical model formulation and junction tree inference in Wiegerinck et al (2006Wiegerinck et al ( , 2007 and approximate inference in van den Broek et al (2008aBroek et al ( , 2008b. In robotics, Theodorou et al (2009Theodorou et al ( , 2010aTheodorou et al ( , 2010b has shown the path integral method has great potential for application.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Optimal control as a graphical model inference problem

2012

View full text Add to dashboard Cite

We reformulate a class of non-linear stochastic optimal control problems introduced by Todorov (in Advances in Neural Information Processing Systems, vol. 19, pp. 1369Systems, vol. 19, pp. -1376Systems, vol. 19, pp. , 2007) as a Kullback-Leibler (KL) minimization problem. As a result, the optimal control computation reduces to an inference computation and approximate inference methods can be applied to efficiently compute approximate optimal controls. We show how this KL control theory contains the path integral control method as a special case. We provide an example of a block stacking task and a multi-agent cooperative game where we demonstrate how approximate inference can be successfully applied to instances that are too complex for exact computation. We discuss the relation of the KL control approach to other inference approaches to control.

show abstract

Section: Discussionmentioning

confidence: 86%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Optimal control as a graphical model inference problem

2012

View full text Add to dashboard Cite

show abstract

“…This generalized the work of Cooper and Shachter to the case of infinite horizons, and cost functions over future states. More recently, this approach has been pursued by applying Bayesian procedures (or minimising Kullback-Leibler divergences) to problems of optimal decision making in MDPs Botvinick and An 2008;Hoffman et al 2009;Toussaint et al 2008).…”

Section: Optimal Control As Inferencementioning

confidence: 99%

Active inference and agency: optimal control without cost functions

2012

View full text Add to dashboard Cite

This paper describes a variational free-energy formulation of (partially observable) Markov decision problems in decision making under uncertainty. We show that optimal control can be cast as active inference. In active inference, both action and posterior beliefs about hidden states minimise a free energy bound on the negative log-likelihood of observed states, under a generative model. In this setting, reward or cost functions are absorbed into prior beliefs about state transitions and terminal states. Effectively, this converts optimal control into a pure inference problem, enabling the application of standard Bayesian filtering techniques. We then consider optimal trajectories that rest on posterior beliefs about hidden states in the future. Crucially, this entails modelling control as a hidden state that endows the generative model with a representation of agency. This leads to a distinction between models with and without inference on hidden control states; namely, agency-free and agency-based models, respectively.

show abstract

“…Following [15], it is possible to move the expectation inside the summation and rewrite the expected utility as…”

Section: Sequential Decision Makingmentioning

confidence: 99%

Inference and Learning for Active Sensing, Experimental Design and Control

Kueck

Hoffman

Doucet

et al. 2009

Pattern Recognition and Image Analysis

View full text Add to dashboard Cite

Abstract. In this paper we argue that maximum expected utility is a suitable framework for modeling a broad range of decision problems arising in pattern recognition and related fields. Examples include, among others, gaze planning and other active vision problems, active learning, sensor and actuator placement and coordination, intelligent humancomputer interfaces, and optimal control. Following this remark, we present a common inference and learning framework for attacking these problems. We demonstrate this approach on three examples: (i) active sensing with nonlinear, non-Gaussian, continuous models, (ii) optimal experimental design to discriminate among competing scientific models, and (iii) nonlinear optimal control. The Principle of Maximum Expected Utility

show abstract

Probabilistic inference for solving discrete and continuous state Markov Decision Processes

Cited by 190 publications

References 5 publications

Optimal control as a graphical model inference problem

Optimal control as a graphical model inference problem

Active inference and agency: optimal control without cost functions

Inference and Learning for Active Sensing, Experimental Design and Control

Contact Info

Product

Resources

About