No abstract

Abstract.We present a general framework for applying machine-learning algorithms to the verification of Markov decision processes (MDPs). The primary goal of these techniques is to improve performance by avoiding an exhaustive exploration of the state space. Our framework focuses on probabilistic reachability, which is a core property for verification, and is illustrated through two distinct instantiations. The first assumes that full knowledge of the MDP is available, and performs a heuristic-driven partial exploration of the model, yielding precise lower and upper bounds on the required probability. The second tackles the case where we may only sample the MDP, and yields probabilistic guarantees, again in terms of both the lower and upper bounds, which provides efficient stopping criteria for the approximation. The latter is the first extension of statistical model-checking for unbounded properties in MDPs. In contrast with other related approaches, we do not restrict our attention to time-bounded (finite-horizon) or discounted properties, nor assume any particular properties of the MDP. We also show how our techniques extend to LTL objectives. We present experimental results showing the performance of our framework on several examples.

Multi-objective probabilistic model checking provides a way to verify several, possibly conflicting, quantitative properties of a stochastic system. It has useful applications in controller synthesis and compositional probabilistic verification. However, existing methods are based on linear programming, which limits the scale of systems that can be analysed and makes verification of time-bounded properties very difficult. We present a novel approach that addresses both of these shortcomings, based on the generation of successive approximations of the Pareto curve for a multi-objective model checking problem. We illustrate dramatic improvements in efficiency on a large set of benchmarks and show how the ability to visualise Pareto curves significantly enhances the quality of results obtained from current probabilistic verification tools.amenable to symbolic (BDD-based) implementations. Value iteration can also be used for time-bounded (finite-horizon) properties, which is impractical with LP. Another alternative is policy iteration, but this is also impractical for timebounded properties, and preliminary investigations in [10] showed no particular improvement over value iteration in the context of probabilistic verification.There has recently been increased interest in multi-objective probabilistic model checking for MDPs [5,9,11,4], which can be used to analyse trade-offs between several, possibly conflicting, quantitative properties. Consider, for example, two events of interest, A and B, and let p σ A and p σ B be the probability that each occurs under an adversary σ of an MDP. In this paper, we study several kinds of multi-objective properties. Achievability queries ask, e.g., "is there an adversary σ satisfying the predicate ψ = p σ A x ∧ p σ B y?" and numerical queries ask, e.g., "what is maximum value of x such that ψ is achievable?". We also consider the Pareto curve of undominated solution points: for this example, the set of pairs (x, y) such that ψ is achievable but any increase in either x or y would necessitate a decrease in the other.Multi-objective techniques have natural applications to controller synthesis for MDPs (e.g., "how can we maximise the probability of successful message transmission, whilst keeping the expected energy usage below 100 mJ?"). They also form the basis of recent compositional verification techniques [15], which decompose model checking into separate tasks for each system component using assume-guarantee reasoning (e.g., "what is the maximum probability of a global system error, under the assumption that component 1 fails with probability at most 0.02?"). This approach has been successfully used to verify probabilistic systems too large to analyse without compositional techniques.Existing multi-objective model checking methods [5,9,11,4] rely on a reduction to LP. The linear program solved, although of a rather different form to the standard (single objective) case, is still linear in the size of the MDP, yielding polynomial time complexity. As discussed above, though, LP-bas...

Abstract. We present a verification framework for analysing multiple quantitative objectives of systems that exhibit both nondeterministic and stochastic behaviour. These systems are modelled as probabilistic automata, enriched with cost or reward structures that capture, for example, energy usage or performance metrics. Quantitative properties of these models are expressed in a specification language that incorporates probabilistic safety and liveness properties, expected total cost or reward, and supports multiple objectives of these types. We propose and implement an efficient verification framework for such properties and then present two distinct applications of it: firstly, controller synthesis subject to multiple quantitative objectives; and, secondly, quantitative compositional verification. The practical applicability of both approaches is illustrated with experimental results from several large case studies.

Abstract. We study two-player stochastic games, where the goal of one player is to satisfy a formula given as a positive boolean combination of expected total reward objectives and the behaviour of the second player is adversarial. Such games are important for modelling, synthesis and verification of open systems with stochastic behaviour. We show that finding a winning strategy is PSPACE-hard in general and undecidable for deterministic strategies. We also prove that optimal strategies, if they exists, may require infinite memory and randomisation. However, when restricted to disjunctions of objectives only, memoryless deterministic strategies suffice, and the problem of deciding whether a winning strategy exists is NP-complete. We also present algorithms to approximate the Pareto sets of achievable objectives for the class of stopping games.