Recursive Markov Decision Processes and Recursive Stochastic Games

Abstract

Abstract. We introduce Recursive Markov Decision Processes (RMDPs) and Recursive Simple Stochastic Games (RSSGs), and study the decidability and complexity of algorithms for their analysis and verification. These models extend Recursive Markov Chains (RMCs), introduced in [EY05a,EY05b] as a natural model for verification of probabilistic procedural programs and related systems involving both recursion and probabilistic behavior. RMCs define a class of denumerable Markov chains with a rich theory generalizing that of stochastic context-free grammars and multi-type branching processes, and they are also intimately related to probabilistic pushdown systems. RMDPs & RSSGs extend RMCs with one controller or two adversarial players, respectively. Such extensions are useful for modeling nondeterministic and concurrent behavior, as well as modeling a system's interactions with an environment. We provide a number of upper and lower bounds for deciding, given an RMDP (or RSSG) A and probability p, whether player 1 has a strategy to force termination at a desired exit with probability at least p. We also address "qualitative" termination questions, where p = 1, and model checking questions.

The Complexity of Stochastic Mueller Games

“…We also present improved memory bounds for randomized strategies. Unlike the results of [10] our results do not extend to infinite state games: for example, the results of [12] showed that even for 2 1 / 2 -player pushdown games optimal strategies need not exist, and for ε > 0 even ε-optimal strategies may require infinite memory. For lower bound of randomized strategies the constructions of [10] do not work: in fact for the family of games used for lower bounds in [10] randomized memoryless almost-sure winning strategies exist.…”

Reachability in recursive Markov decision processes

“…For games with finitely many vertices, the corresponding decision algorithms have been designed [9,2,1] and also implemented in verifications tools such as PRISM (see, e.g., [10]). Recently, the scope of this study has been extended to a class of infinite-state games generated by recursive state machines (RSM) [6,7]. Intuitively, a RSM is a finite collection of finite-state automata which can call each other in a recursive fashion, maintaining the (unbounded) stack of activation records.…”

The 10,000 Facets of MDP Model Checking

“…MDPs with the energy condition can also be seen as infinite-state MDPs that operate with a counter [36], or equivalently, with stacks over an unary stack alphabet, which again is closely related to the model of recursive MDPs [35,73]. Although reasoning about temporal properties with weight constraints is in general undecidable, even in the non-probabilistic case [29], the maximal or minimal probabilities for LTL formulas with constraints for the weight accumulated in windows of a fixed length are computable using a reduction to standard LTL [19].…”

“…Model checking of (discrete-time) uncountable MDPs is treated in [144]. Countably infinite variants of MDPs include probabilistic lossy channel systems [7] where message losses have a probabilistic behavior while the component finite-state processes behave nondeterministically, one-counter MDPs [36], MDPs equipped with counters that can be arbitrarily negative or positive, and recursive MDPs [71,73] (that subsume onecounter MDPs). Recursive MDPs are equivalent to push-down MDPs.…”