A Practitioner’s Guide to MDP Model Checking Algorithms

Abstract

Model checking undiscounted reachability and expected-reward properties on Markov decision processes (MDPs) is key for the verification of systems that act under uncertainty. Popular algorithms are policy iteration and variants of value iteration; in tool competitions, most participants rely on the latter. These algorithms generally need worst-case exponential time. However, the problem can equally be formulated as a linear program, solvable in polynomial time. In this paper, we give a detailed overview of today’s state-of-the-art algorithms for MDP model checking with a focus on performance and correctness. We highlight their fundamental differences, and describe various optimizations and implementation variants. We experimentally compare floating-point and exact-arithmetic implementations of all algorithms on three benchmark sets using two probabilistic model checkers. Our results show that (optimistic) value iteration is a sensible default, but other algorithms are preferable in specific settings. This paper thereby provides a guide for MDP verification practitioners—tool builders and users alike.

PET – A Partial Exploration Tool for Probabilistic Verification

“…Details on Model Structure. Reiterating [6,39,24], we again underline that the structure of a model is the leading factor for the (relative) performance of algorithmic approaches. This is supported by our comparison of CE vs. PE vs. hybrid approaches above.…”

“…It is well known that the structure of a model is the determining factor for the relative performance of different algorithmic approaches, see e.g. [6,39,24], in particular far more than the number of states or transitions. (This is also supported by our comparison of CE and PE in App.…”

“…Note that the classical approaches strategy iteration and quadratic programming are sound in theory, but (i) the available implementations of [33] are prototypical and unsound [33,Sec. 5], and (ii) these approaches usually either are not practically efficient or use heuristics that actually make them unsound [24].…”