Improved Strong Worst-case Upper Bounds for MDP Planning

Abstract

The Markov Decision Problem (MDP) plays a central role in AI as an abstraction of sequential decision making. We contribute to the theoretical analysis of MDP planning, which is the problem of computing an optimal policy for a given MDP. Specifically, we furnish improved strong worstcase upper bounds on the running time of MDP planning. Strong bounds are those that depend only on the number of states n and the number of actions k in the specified MDP; they have no dependence on affiliated variables such as the discount factor and the number of bits needed to represent the MDP. Worst-case bounds apply to every run of an algorithm; randomised algorithms can typically yield faster expected running times. While the special case of 2-action MDPs (that is, k = 2) has recently received some attention, bounds for general k have remained to be improved for several decades. Our contributions are to this general case. For k ≥ 3, the tightest strong upper bound shown to date for MDP planning belongs to a family of algorithms called Policy Iteration. This bound is only a polynomial improvement over a trivial bound of poly(n, k) · k n [Mansour and Singh, 1999]. In this paper, we generalise a contrasting algorithm called the Fibonacci Seesaw, and derive a bound of poly(n, k) · k 0.6834n . The key construct that we use is a template to map algorithms for the 2-action setting to the general setting. Interestingly, this idea can also be used to design Policy Iteration algorithms with a running time upper bound of poly(n, k)·k 0.7207n . Both our results improve upon bounds that have stood for several decades.

Lexicographic refinements in stationary possibilistic Markov Decision Processes

“…The convergence of these algorithms has been shown and their efficiency has been assessed experimentally. In particular, the algorithm of Policy Iteration appears to be experimentally faster than the algorithm of Bounded Utility Value Iteration (which is also usual in the stochastic case [16]).…”

“…When the horizon of the MDP is not finite, equations (16) and (17) are not enough to rank-order the policies. The length of the trajectories may be infinite, as well as their number.…”

“…Policy iteration (Algorithm 5) converges and is guaranteed to find a policy optimal for the (l, c) lexicographic criterion in finite time and usually in a few iterations. As for the algorithmic complexity of the classical, stochastic, policy iteration algorithm (which is still not well understood [16]), a tight bound worst-case complexity of lexicographic policy iteration is hard to obtain. Therefore, we provide an upper-bound of this complexity.…”