Reachability in recursive Markov decision processes

Brázdil, Tomǎš; Brožek, Václav; Forejt, Vojtěch; Kučera, Antonín

doi:10.1016/j.ic.2007.09.002

Cited by 29 publications

(43 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We say that a strategy is static if for each type T i controlled by that player the strategy always chooses the same action a i , or the same probability distribution on actions, for all entities of type T i in all histories. 4 Our notion of an arbitrary strategy is quite general (it can depend on all the details of the entire history, and be randomized, etc.). However, it was shown in [14] that for the objective of optimizing extinction probability, both players have optimal static strategies in BSSGs.…”

Section: Definitions and Backgroundmentioning

confidence: 99%

Greatest fixed points of probabilistic min/max polynomial equations, and reachability for branching Markov decision processes

Etessami

Stewart

Yannakakis

2018

Information and Computation

View full text Add to dashboard Cite

We give polynomial time algorithms for quantitative (and qualitative) reachability analysis for Branching Markov Decision Processes (BMDPs). Specifically, given a BMDP, and given an initial population, where the objective of the controller is to maximize (or minimize) the probability of eventually reaching a population that contains an object of a desired (or undesired) type, we give algorithms for approximating the supremum (infimum) reachability probability, within desired precision ǫ > 0, in time polynomial in the encoding size of the BMDP and in log(1/ǫ). We furthermore give P-time algorithms for computing ǫ-optimal strategies for both maximization and minimization of reachability probabilities. We also give P-time algorithms for all associated qualitative analysis problems, namely: deciding whether the optimal (supremum or infimum) reachability probabilities are 0 or 1. Prior to this paper, approximation of optimal reachability probabilities for BMDPs was not even known to be decidable.Our algorithms exploit the following basic fact: we show that for any BMDP, its maximum (minimum) non-reachability probabilities are given by the greatest fixed point (GFP) solution g * ∈ [0, 1] n of a corresponding monotone max (min) Probabilistic Polynomial System of equations (max/minPPS), x = P (x), which are the Bellman optimality equations for a BMDP with non-reachability objectives. We show how to compute the GFP of max/minPPSs to desired precision in P-time.We also study more general branching simple stochastic games (BSSGs) with (non-)reachability objectives. We show that: (1) the value of these games is captured by the GFP, g * , of a corresponding max-minPPS, x = P (x); (2) the quantitative problem of approximating the value is in TFNP; and (3) the qualitative problems associated with the value are all solvable in P-time.

show abstract

Section: Definitions and Backgroundmentioning

confidence: 99%

Greatest fixed points of probabilistic min/max polynomial equations, and reachability for branching Markov decision processes

Etessami

Stewart

Yannakakis

2018

Information and Computation

View full text Add to dashboard Cite

show abstract

“…Also observe that A.2 does not hold for infinite-state MDPs (a counterexample is simple to construct even for a single reachability objective, see e.g. [5,Example 6]). …”

Section: Expectation Objectivesmentioning

confidence: 99%

Markov Decision Processes with Multiple Long-run Average Objectives

Brázdil¹,

Brožek²,

Chatterjee³

et al. 2014

Logical Methods in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. We study Markov decision processes (MDPs) with multiple limit-average (or mean-payoff) functions. We consider two different objectives, namely, expectation and satisfaction objectives. Given an MDP with k k k limit-average functions, in the expectation objective the goal is to maximize the expected limit-average value, and in the satisfaction objective the goal is to maximize the probability of runs such that the limit-average value stays above a given vector. We show that under the expectation objective, in contrast to the case of one limit-average function, both randomization and memory are necessary for strategies even for ε ε ε-approximation, and that finite-memory randomized strategies are sufficient for achieving Pareto optimal values. Under the satisfaction objective, in contrast to the case of one limit-average function, infinite memory is necessary for strategies achieving a specific value (i.e. randomized finite-memory strategies are not sufficient), whereas memoryless randomized strategies are sufficient for ε ε ε-approximation, for all ε > 0 ε > 0 ε > 0. We further prove that the decision problems for both expectation and satisfaction objectives can be solved in polynomial time and the trade-off curve (Pareto curve) can be ε ε ε-approximated in time polynomial in the size of the MDP and, and exponential in the number of limitaverage functions, for all ε > 0 ε > 0 ε > 0. Our analysis also reveals flaws in previous work for MDPs with multiple mean-payoff functions under the expectation objective, corrects the flaws, and allows us to obtain improved results. ACM CCS: [Mathematics

show abstract

“…The result about termination has been extended to general qualitative reachability in [13], even for a more general model of Markov decision processes generated by BPA. Thus, we obtain the following:…”

Section:   mentioning

confidence: 99%