Optimization Models for the First Arrival Target Distribution Function in Discrete Time

Yu, Stella X.; Lin, You-Wei; Yan, Pingfan

doi:10.1006/jmaa.1998.6015

Cited by 40 publications

(32 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The analysis of single-objective (cost-bounded) reachability in MDPs is an active area of research in both AI and formal method communities, and referred to in, e.g., [18,35,48]. Various model checking approaches for single objectives exist.…”

Section: Introductionmentioning

confidence: 99%

Multi-cost Bounded Reachability in MDP

Hartmanns

Junges

Katoen

et al. 2018

Tools and Algorithms for the Construction and Analysis of Systems

View full text Add to dashboard Cite

Abstract. We provide an efficient algorithm for multi-objective modelchecking problems on Markov decision processes (MDPs) with multiple cost structures. The key problem at hand is to check whether there exists a scheduler for a given MDP such that all objectives over cost vectors are fulfilled. Reachability and expected cost objectives are covered and can be mixed. Empirical evaluation shows the algorithm's scalability. We discuss the need for output beyond Pareto curves and exploit the available information from the algorithm to support decision makers.

show abstract

Section: Introductionmentioning

confidence: 99%

Multi-cost Bounded Reachability in MDP

Hartmanns

Junges

Katoen

et al. 2018

Tools and Algorithms for the Construction and Analysis of Systems

View full text Add to dashboard Cite

show abstract

“…al. [57] consider an absorbing Markov process with finite state and control spaces. The absorbing states are not cost-free and there exist one-time terminal rewards, therefore, this problem is not completely related to one that we work on.…”

Section: Howard and Mathesonmentioning

confidence: 99%

Risk-Averse Control of Undiscounted Transient Markov Models

Çavuş¹,

Ruszczyński²

2014

SIAM J. Control Optim.

View full text Add to dashboard Cite

The classical optimal control problems for discrete-time, transient Markov processes are infinite horizon, undiscounted expected total cost or reward models. Some examples of these models are optimal stopping problems and stochastic shortest or longest path problems, which may have applications in health-care, finance, and maintenance. However, such expected value models implicitly assume the decision maker is risk-neutral, so they may not be appropriate for several real-life problems.In this study, we use Markov risk measures to formulate a risk-averse version of the optimal control problem for transient Markov processes with general state and compact control spaces. We derive risk-averse dynamic programming equations and show that they have a unique solution which is also the optimal value of the Markov control problem. Furthermore, it is shown that a randomized policy may be strictly better than deterministic policies, when risk measures are employed.We suggest two algorithms, value iteration and policy iteration methods, for solving the dynamic programming equations and show their convergence. In general, each policy evaluation step of the policy iteration algorithm requires solving a system of nonsmooth equations. We use a version of nonsmooth Newton method to solve these equations and show its global convergence.ii We further consider a risk-averse finite horizon Markov control problem under randomized policies and derive a value iteration method for its solution.Finally, we work on asset selling, organ transplant, and credit card examples to illustrate the theory for infinite horizon problem, and present numerical results.iii

show abstract

“…Applications of such models include equipment maintenance, queuing systems, reliability engineering, risk analysis and so on, in which the target set corresponds to the failure, refusal or down states and the first passage time to the target set denotes the working time or functioning life of the system. As is well-known, first passage models have been studied by several authors, such as Derman [1] , Lin [8] , Lin, Tomkins and Wang [9] , Liu and Huang [11] , Liu and Liu [12,13] , Ohtsubo [14] , and Yu, Lin and Yan [18] . In general, the main goal of this kind of models is to maximize the expected total reward obtained during a first passage time to some target set.…”

Section: Introductionmentioning

confidence: 99%

First passage models for denumerable semi-Markov decision processes with nonnegative discounted costs

Huang

Guo

2011

Acta Math. Appl. Sin. Engl. Ser.

View full text Add to dashboard Cite

This paper considers a first passage model for discounted semi-Markov decision processes with denumerable states and nonnegative costs. The criterion to be optimized is the expected discounted cost incurred during a first passage time to a given target set. We first construct a semi-Markov decision process under a given semi-Markov decision kernel and a policy. Then, we prove that the value function satisfies the optimality equation and there exists an optimal (or ε-optimal) stationary policy under suitable conditions by using a minimum nonnegative solution approach. Further we give some properties of optimal policies. In addition, a value iteration algorithm for computing the value function and optimal policies is developed and an example is given. Finally, it is showed that our model is an extension of the first passage models for both discrete-time and continuous-time Markov decision processes.

show abstract

Optimization Models for the First Arrival Target Distribution Function in Discrete Time

Cited by 40 publications

References 11 publications

Multi-cost Bounded Reachability in MDP

Multi-cost Bounded Reachability in MDP

Risk-Averse Control of Undiscounted Transient Markov Models

First passage models for denumerable semi-Markov decision processes with nonnegative discounted costs

Contact Info

Product

Resources

About