Discrete Dynamic Programming with Sensitive Discount Optimality Criteria

We prove that the classic policy-iteration method (Howard 1960), including the Simplex method (Dantzig 1947) with the most-negative-reduced-cost pivoting rule, is a strongly polynomial-time algorithm for solving the Markov decision problem (MDP) with a fixed discount rate. Furthermore, the computational complexity of the policyiteration method (including the Simplex method) is superior to that of the only known strongly polynomial-time interior-point algorithm ([28] 2005) for solving this problem. The result is surprising since the Simplex method with the same pivoting rule was shown to be exponential for solving a general linear programming (LP) problem, the Simplex (or simple policy-iteration) method with the smallest-index pivoting rule was shown to be exponential for solving an MDP regardless of discount rates, and the policy-iteration method was recently shown to be exponential for solving a undiscounted MDP. We also extend the result to solving MDPs with sub-stochastic and transient state transition probability matrices.

show abstract

“…Veinott [27]. Note that the inverse of (I − P ) has the expansion form (I − P ) −1 = I + P + P 2 + .…”

Section: Extensions and Remarksmentioning

confidence: 99%

The Simplex and Policy-Iteration Methods Are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount Rate

2011

Mathematics of OR

144

168

View full text Add to dashboard Cite

show abstract

“…However, some of the techniques used here are, perhaps, somewhat more in the spirit of the recent developments such as those reported in Avrachenkov et al [9,10]. More particularly, in the context of perturbation and sensitivity analysis of MDP's the papers by Schweitzer [65,66] and Veinott [68] are quite relevant to the results reported here.…”

Section: The Asymmetric Quadratic Perturbationmentioning

confidence: 76%

Controlled Markov Chains, Graphs, and Hamiltonicity

Filar

2006

FNT in Stochastic Systems

View full text Add to dashboard Cite

“…The restriction to stationary nonrandomized policies is justified because Hypothesis C has been shown to suffice for such a policy to be optimal over the class of all history-remembering policies, see [5] or [21]. Further, such a policy can be found by linear programming, by policy improvement, or by successive approximation.…”

Section: A Sequential Decision Processmentioning

confidence: 99%

“…The fact that the risk-averse exponential utility function has u(0) = −1 and the weakened hypothesis can be used to work around this difficulty by ruling out any stationary policy that plays a bandit at each state in any closed communicating class. A second difficulty arises from the fact that the interchange argument in Proposition 5.1 can no longer rest on the classic results in [5] or [21]. The interested reader is referred to the analysis in [6] and to the characterization of optimal policies in [7].…”

Section: Relaxing Hypothesis Cmentioning

confidence: 99%

The multi-armed bandit, with constraints

2012

View full text Add to dashboard Cite

The early sections of this paper present an analysis of a Markov decision model that is known as the multi-armed bandit under the assumption that the utility function of the decision maker is either linear or exponential. The analysis includes efficient procedures for computing the expected utility associated with the use of a priority policy and for identifying a priority policy that is optimal. The methodology in these sections is novel, building on the use of elementary row operations. In the later sections of this paper, the analysis is adapted to accommodate constraints that link the bandits.

show abstract

Discrete Dynamic Programming with Sensitive Discount Optimality Criteria

Cited by 240 publications

References 23 publications

The Simplex and Policy-Iteration Methods Are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount Rate

The Simplex and Policy-Iteration Methods Are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount Rate

Controlled Markov Chains, Graphs, and Hamiltonicity

The multi-armed bandit, with constraints

Contact Info

Product

Resources

About