A Modified Dynamic Programming Method for Markovian Decision Problems

MacQueen, James B.

doi:10.21236/ad0619764

Cited by 11 publications

(16 citation statements)

References 3 publications

(5 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Unfortunately, although most likely neither the true optimal percentile policy nor the approximate one put positive weight on these state-action pairs, Theorem 4 states that our confidence in the approximate policy should depend on this reduced number of transition observations from the given pairs. We apply the idea of action elimination, proposed by MacQueen (1966) in the context of the nominal MDP, to the percentile optimization framework to relax this dependence.…”

Section: Improving the Bound With Action Eliminationmentioning

confidence: 99%

Percentile Optimization for Markov Decision Processes with Parameter Uncertainty

Delage

Mannor

2010

Operations Research

163

125

View full text Add to dashboard Cite

show abstract

Section: Improving the Bound With Action Eliminationmentioning

confidence: 99%

Percentile Optimization for Markov Decision Processes with Parameter Uncertainty

Delage

Mannor

2010

Operations Research

163

125

View full text Add to dashboard Cite

show abstract

“…In Section VI, the performance of our approximation scheme and Gauss-Seidel VI is compared numerically. The RVI algorithm, proposed by [4] for average reward problems and generalized to discounted reward settings by [5] and [6], is shown to have a (αβ RVI ) k convergence in [7]. The convergence is proved in terms of the relative value function, which is the difference between the value of each state and the value of a fixed pre-selected state, and β RVI < 1 is the second largest eigenvalue of the transition probability matrix corresponding to the optimal policy.…”

Section: A Related Literaturementioning

confidence: 99%

Weighted difference approximation of value functions for slow-discounting Markov Decision Processes

Chow¹,

Qin²

2014

53rd IEEE Conference on Decision and Control

View full text Add to dashboard Cite

Abstract-Modern applications of the theory of Markov Decision Processes (MDPs) often require frequent decision making, that is, taking an action every microsecond, second, or minute. Infinite horizon discount reward formulation is still relevant for a large portion of these applications, because actual time span of these problems can be months or years, during which discounting factors due to e.g. interest rates are of practical concern. In this paper, we show that, for such MDPs with discount rate α close to 1, under a common ergodicity assumption, a weighted difference between two successive value function estimates obtained from the classical value iteration (VI) is a better approximation than the value function obtained directly from VI. Rigorous error bounds are established which in turn show that the approximation converges to the actual value function in a rate (αβ) k with β < 1. This indicates a geometric convergence even if discount factor α → 1. Furthermore, we explicitly link the convergence speed to the system behaviors of the MDP using the notion of ǫ−mixing time and extend our result to Q-functions. Numerical experiments are conducted to demonstrate the convergence properties of the proposed approximation scheme.

show abstract

“…Such an implementation would need to use the entire node space or a layered-network approach. (We also attempted to use action elimination (MacQueen 1966), a technique that requires both an upper and a lower bound; however, its performance was not significantly better than dynamic programming. We used upper and lower bounds that were multiples of Euclidean distance.…”

Section: Bander and White Heuristic Search For Stochastic Shortest Pathsmentioning

confidence: 99%

A Heuristic Search Approach for a Nonstationary Stochastic Shortest Path Problem with Terminal Cost

Bander

White

2002

Transportation Science

View full text Add to dashboard Cite

We present a best-first heuristic search approach for determining an optimal policy for a stochastic shortest path problem. A vehicle is to travel from an origin, starting at time t0, to a destination, where once the destination is reached a terminal cost is accrued. The terminal cost depends on the time of arrival. Travel time along each arc in the network is modeled as a random variable with a distribution dependent on the time that travel along the arc is begun. The objective is to determine a routing policy that minimizes expected total cost. A routing policy is a rule that assigns the next arc to traverse, given the current node and time. The heuristic search algorithm that we specialize to this stochastic problem is AO*. We demonstrate the significant computational advantages of AO*, relative to dynamic programming, on the basis of run time and storage, using a 131-intersection network of the major roads in Cleveland, Ohio. Further computational experience is based on grid networks that were randomly generated to have characteristics similar to transportation networks, and on randomly generated unstructured networks.

show abstract

A Modified Dynamic Programming Method for Markovian Decision Problems

Cited by 11 publications

References 3 publications

Percentile Optimization for Markov Decision Processes with Parameter Uncertainty

Percentile Optimization for Markov Decision Processes with Parameter Uncertainty

Weighted difference approximation of value functions for slow-discounting Markov Decision Processes

A Heuristic Search Approach for a Nonstationary Stochastic Shortest Path Problem with Terminal Cost

Contact Info

Product

Resources

About