Generalizing Markov decision processes to imprecise probabilities

Harmanec, David

doi:10.1016/s0378-3758(01)00210-5

Cited by 22 publications

(24 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Harmanec [7] studied a similar problem where the imprecision was defined in the transition probabilities, rather than the rewards. However, a decision maker can only implement a single policy in practice.…”

Section: Introductionmentioning

confidence: 99%

Sensitivity Analysis in Markov Decision Processes with Uncertain Reward Parameters

Tan

Hartman

2011

Journal of Applied Probability

View full text Add to dashboard Cite

Sequential decision problems can often be modeled as Markov decision processes. Classical solution approaches assume that the parameters of the model are known. However, model parameters are usually estimated and uncertain in practice. As a result, managers are often interested in how estimation errors affect the optimal solution. In this paper we illustrate how sensitivity analysis can be performed directly for a Markov decision process with uncertain reward parameters using the Bellman equations. In particular, we consider problems involving (i) a single stationary parameter, (ii) multiple stationary parameters, and (iii) multiple nonstationary parameters. We illustrate the applicability of this work through a capacitated stochastic lot-sizing problem.

show abstract

Section: Introductionmentioning

confidence: 99%

Sensitivity Analysis in Markov Decision Processes with Uncertain Reward Parameters

Tan

Hartman

2011

Journal of Applied Probability

View full text Add to dashboard Cite

show abstract

“…Two solutions have been suggested and studied in the literature: (i) learning-update the transition probabilities as we observe transitions [4,5], and (ii) sets-only assume the transition probabilities belong to some convex set [5,7,2,3].…”

Section: Introductionmentioning

confidence: 99%

“…One notable exception is in [3], where an algorithm is suggested to find the set of all maximal elements with respect to a partial preference order, based on comparing intervals. In that way, not only the extreme solutions are recovered.…”

Section: Introductionmentioning

confidence: 99%

“…In that way, not only the extreme solutions are recovered. However, in [3] it is not questioned in what sense the proposed dynamic programming method leads to optimal policies. In this article we approach the problem from a more logical side: we first define a notion of optimality and investigate whether the dynamic programming argument holds for this notion of optimality, instead of blindly "generalising" Bellman's algorithm.…”

Section: Introductionmentioning

confidence: 99%

“…Unfortunately, the first condition does not hold when using the partial order of [3] (see counterexample in [1]). Hence, the algorithm of [3] actually does not result in optimal control laws in the sense of maximality with respect to the suggested order.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning and Optimal Control of Imprecise Markov Decision Processes by Dynamic Programming Using the Imprecise Dirichlet Model

Troffaes

2004

Soft Methodology and Random Information Systems

View full text Add to dashboard Cite

Summary. In this paper, we investigate the conditions under which dynamic programming yields a solution to simultaneous learning and optimal control of a Markov decision process. First, we introduce a new optimality criterion that allows act-state dependence. This criterion is based on a partial preference ordering induced by an imprecise probability model of the dynamics of the system, updated by observations of the state and control history of the system. Then, we show that dynamic programming yields the set of all optimal solutions if the imprecise probability model satisfies particular properties. When we model learning of the system dynamics by an imprecise Dirichlet model, these properties turn out to be satisfied.

show abstract

Sensitivity Analysis and Dynamic Programming

Tan

Hartman

2011

Wiley Encyclopedia of Operations Research and Management Science

View full text Add to dashboard Cite

The validity of any dynamic programming solution depends on the accuracy of the model and the data for a given instance. However, the parameters of the model are often uncertain and estimated in practice. In this paper, we discuss various concepts of sensitivity analysis and survey the work on sensitivity analysis in the dynamic programming literature. In addition, we highlight the relationship between dynamic programming and linear programming and how the approaches and results from the latter can be applied to dynamic programs. A list of promising areas of future research is provided.

show abstract

Generalizing Markov decision processes to imprecise probabilities

Cited by 22 publications

References 7 publications

Sensitivity Analysis in Markov Decision Processes with Uncertain Reward Parameters

Sensitivity Analysis in Markov Decision Processes with Uncertain Reward Parameters

Learning and Optimal Control of Imprecise Markov Decision Processes by Dynamic Programming Using the Imprecise Dirichlet Model

Sensitivity Analysis and Dynamic Programming

Contact Info

Product

Resources

About