On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference

Abstract

Abstract-We present a reformulation of the stochastic optimal control problem in terms of KL divergence minimisation, not only providing a unifying perspective of previous approaches in this area, but also demonstrating that the formalism leads to novel practical approaches to the control problem. Specifically, a natural relaxation of the dual formulation gives rise to exact iterative solutions to the finite and infinite horizon stochastic optimal control problem, while direct application of Bayesian inference methods yields instances of risk sensitive control. We furthermore study corresponding formulations in the reinforcement learning setting and present model free algorithms for problems with both discrete and continuous state and action spaces. Evaluation of the proposed methods on the standard Gridworld and Cart-Pole benchmarks verifies the theoretical insights and shows that the proposed methods improve upon current approaches.

Active inference and agency: optimal control without cost functions

“…Related work on stochastic optimal control (Kappen 2005a,b;van den Broek et al 2008;Rawlik et al 2010), exploits the reduction of control problems to inference problems by appealing to variational techniques to provide efficient and computationally tractable solutions. In particular, formulating the problem in terms of Kullback-Leibler minimization (Kappen 2005a,b) and path integrals of cost functions using the Feynman-Kac formula (Theodorou et al 2010;Braun et al 2011).…”

Bounded Rationality, Abstraction, and Hierarchical Decision-Making: An Information-Theoretic Optimality Principle

“…In this study, we use an information-theoretic model of bounded rational decision-making Braun, 2012, 2013;Braun and Ortega, 2014; that has precursors in the economic literature (McKelvey and Palfrey, 1995;Mattsson and Weibull, 2002;Sims, 2003Sims, , 2005Sims, , 2006Sims, , 2010Wolpert, 2006) and that is closely related to recent advances in the information theory of perception-action systems (Todorov, 2007(Todorov, , 2009Still, 2009;Friston, 2010;Peters et al, 2010;Tishby and Polani, 2011;Daniel et al, 2012Daniel et al, , 2013Kappen et al, 2012;Rawlik et al, 2012;Rubin et al, 2012;Neymotin et al, 2013;Tkačik and Bialek, 2014;Palmer et al, 2015). The basis of this approach is formalized by a free energy principle that trades off expected utility, and the cost of computation that is required to adapt the system accordingly in order to achieve high utility.…”

Generalizing Movements with Information-Theoretic Stochastic Optimal Control

“…We will first review these two approaches and point out important differences to our approach. Subsequently, we will discuss SOC control algorithms that also use Kullback-Leibler divergence terms to determine the policy, e.g., dynamic policy programming [2], SOC by approximate inference [23] and path integral approaches [27,24]. We will also discuss the relation to approximate dynamic programming with soft-max operators [19] and existing policy search algorithms [8].…”

“…The goal of Stochastic Optimal Control (SOC), see [26,12,31,23], is to find the optimal control for a finite horizon of time steps. A common approach to solve the SOC problem is dynamic programming, i.e.…”

“…As the data-generating distribution incorporates the exploration of the previous policy, the new policy will continue to explore in a similar area of the state space as the old policy. Recent SOC approaches [23,2] penalize the deviation of the new policy from the previous policy, instead of penalizing the deviation of the resulting state distributions. While such an approach continues to explore in a similar action space as the previous policy, it does not limit the change of the state distributions.…”