Greatest fixed points of probabilistic min/max polynomial equations, and reachability for branching Markov decision processes

Abstract

We give polynomial time algorithms for quantitative (and qualitative) reachability analysis for Branching Markov Decision Processes (BMDPs). Specifically, given a BMDP, and given an initial population, where the objective of the controller is to maximize (or minimize) the probability of eventually reaching a population that contains an object of a desired (or undesired) type, we give algorithms for approximating the supremum (infimum) reachability probability, within desired precision ǫ > 0, in time polynomial in the encoding size of the BMDP and in log(1/ǫ). We furthermore give P-time algorithms for computing ǫ-optimal strategies for both maximization and minimization of reachability probabilities. We also give P-time algorithms for all associated qualitative analysis problems, namely: deciding whether the optimal (supremum or infimum) reachability probabilities are 0 or 1. Prior to this paper, approximation of optimal reachability probabilities for BMDPs was not even known to be decidable.Our algorithms exploit the following basic fact: we show that for any BMDP, its maximum (minimum) non-reachability probabilities are given by the greatest fixed point (GFP) solution g * ∈ [0, 1] n of a corresponding monotone max (min) Probabilistic Polynomial System of equations (max/minPPS), x = P (x), which are the Bellman optimality equations for a BMDP with non-reachability objectives. We show how to compute the GFP of max/minPPSs to desired precision in P-time.We also study more general branching simple stochastic games (BSSGs) with (non-)reachability objectives. We show that: (1) the value of these games is captured by the GFP, g * , of a corresponding max-minPPS, x = P (x); (2) the quantitative problem of approximating the value is in TFNP; and (3) the qualitative problems associated with the value are all solvable in P-time.

A Polynomial Time Algorithm for Computing Extinction Probabilities of Multitype Branching Processes

“…Since the publication of the conference version of this paper [19] 9 , we have obtained several subsequent results in papers that build on and extend this work. Specifically, in [18] and [20] we have studied more general branching Markov decision processes, which extend BPs with a controller that can take actions to influence the reproduction probabilities with the goal of maximizing or minimizing the extinction probability, and we studied also their associated max/min probabilistic polynomial systems of equations (max/minPPSs) which involve also a max or min operator. We have shown that a Generalized Newton's method, which involves linear programming in each iteration, can be used to compute both the least and greatest fixed point solutions of such systems of equations in polynomial time to desired precision (in the standard Turing model of computation).…”

Recursive stochastic games with positive rewards

“…(See also [3], where the two-player stochastic game version of qualitative reachability problems was considered.) We remark that in the case of BMDPs, the supremum reachability probability is 1 if and only if there is a strategy that achieves it, and this can be decided in polynomial time [14]. However, note that the equivalence between 1-RMDP and BMDP with respect to the extinction probability does not carry over to the reachability probability, for essentially the same reason that it does not hold in the reward model with 0 rewards.…”

“…Players can use whatever strategy they wish to optimize a given objective. See [17,13,14] for more on these models. We mention that the results of this paper yield directly a polynomial time algorithm, given a BMDP, for computing the optimal (maximum or minimum) expected number of descendants of a given type for an object of a given type.…”

“…Two (equivalent) purely probabilistic recursive models, Recursive Markov chains and probabilistic Pushdown Systems (pPDSs) were introduced in [16] and [11], and have been studied in several papers subsequently. These models were extended to the optimization and game setting of (1)-RMDPs and (1)-RSSGs in [17], and studied further in [2,3,13,14]. As mentioned earlier, the games considered in these earlier papers had the goal of maximizing/minimizing termination or reachability probability, which can be irrational.…”

“…The problem of approximating the optimal termination probabilities for 1-RMDPs (and BMDPs) was addressed in [13], which gave efficient algorithms for computing approximately the optimal termination probabilities of 1-RMDPs within any desired accuracy in polynomial time in the size of the given 1-RMDP and the number of bits of precision, and for computing -optimal strategies. Efficient algorithms for approximately optimizing the reachability probabilities of BMDPs in polynomial time were presented in [14] (these algorithms do not apply however to reachability analysis of 1-RMDPs). Our focus in this paper however is on exact optimization, and the objective is based on rewards rather than termination/reachability probability.…”