The Simplex and Policy-Iteration Methods Are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount Rate

Abstract

We prove that the classic policy-iteration method (Howard 1960), including the Simplex method (Dantzig 1947) with the most-negative-reduced-cost pivoting rule, is a strongly polynomial-time algorithm for solving the Markov decision problem (MDP) with a fixed discount rate. Furthermore, the computational complexity of the policyiteration method (including the Simplex method) is superior to that of the only known strongly polynomial-time interior-point algorithm ([28] 2005) for solving this problem. The result is surprising since the Simplex method with the same pivoting rule was shown to be exponential for solving a general linear programming (LP) problem, the Simplex (or simple policy-iteration) method with the smallest-index pivoting rule was shown to be exponential for solving an MDP regardless of discount rates, and the policy-iteration method was recently shown to be exponential for solving a undiscounted MDP. We also extend the result to solving MDPs with sub-stochastic and transient state transition probability matrices.

On Simplex Pivoting Rules and Complexity Theory

“…On another front, the interesting polynomial simplex-like algorithm of Kelner and Spielman [20] does not settle Conjecture 2 because it is weakly polynomial, as the complexity of each iteration, and the number of iterations, depends (polynomially of course) on the bits of the integers in the input. Other recent results related to Conjecture 1 can be found in [6,35].…”

Simplex Algorithm for Countable-State Discounted Markov Decision Processes

“…By Lemma 4.12, we know thatδ(s, a, N (u k )) → 0 as k → ∞. We will also show that γ u k ,N (u k ) (s, a) becomes nonpositive as k → ∞, which will contradict (25), and thus, we will conclude thatȳ is feasible to (P).…”

“…Recently, complexity of the simplex method with the Dantzig's pivoting rule for finite-state MDPs was studied in [25]. If one can derive a number of iterations (or computational complexity) for the simplex algorithm for countable-state MDPs to find a policy whose value function is within a given threshold from the optimal value function, then it would be possible to compare the convergence rates of the algorithms for countable-state MDPs by comparing the result for the simplex algorithm to the ones in [23,21].…”

“…It is well known that policy iteration, one of the popular solution methods for MDPs with finite state space, can be viewed as the simplex method applied to an equivalent LP formulation of the MDP. A recent result in [25] showed that for finite-state MDPs, simplex method with Dantzig's pivoting rule (for maximization, choosing a non-basic variable with the most positive reduced cost) is strongly polynomial for a fixed discount factor, and the complexity bound is better than that of the other solution methods.…”

Comments on: Recent progress on the combinatorial diameter of polytopes and simplicial complexes

“…It should be remarked that the diameter of the resulting polytopes is actually smaller than the Hirsch bound. Curiously, Ye (2011) showed that the simplex method using Dantzig's pivot rule (where one chooses the entering variable with the largest reduced cost coefficient). is strongly polynomial for the linear programs derived from Markov Decision Processes with Fixed Discount (which is not the setting for the other papers, but is an important case of MDPs).…”