Optimistic Value Iteration

Abstract

Markov decision processes are widely used for planning and verification in settings that combine controllable or adversarial choices with probabilistic behaviour. The standard analysis algorithm, value iteration, only provides lower bounds on infinite-horizon probabilities and rewards. Two "sound" variations, which also deliver an upper bound, have recently appeared. In this paper, we present a new sound approach that leverages value iteration's ability to usually deliver good lower bounds: we obtain a lower bound via standard value iteration, use the result to "guess" an upper bound, and prove the latter's correctness. We present this optimistic value iteration approach for computing reachability probabilities as well as expected rewards. It is easy to implement and performs well, as we show via an extensive experimental evaluation using our implementation within the mcsta model checker of the Modest Toolset.

The probabilistic model checker Storm

“…One way to combat these problems is to approach the solution from both directions, a technique referred to as interval iteration [15,23,58]. Storm implements the latter and additionally the more recent sound value iteration [110] and optimistic value iteration [71]. Numerical errors aside 4 , these methods ensure a correct result within a user-defined accuracy and come with a small time penalty as shown in Sect.…”

Multi-objective Optimization of Long-run Average and Total Rewards

“…The maximal total reward in M can be computed using standard techniques such as value iteration and policy iteration [46] as well as the more recent sound value iteration and optimistic value iteration [48,36]. The latter two provide sound precision guarantees for the output value…”

Multi-objective Optimization of Long-run Average and Total Rewards

“…The supremum sup {Ex σ (lra(R w )) | σ ∈ Σ } is attained by some memoryless deterministic strategy σ w ∈ Σ md [30]. Such a strategy and the induced value v w = Ex σw (lra(R w )) can be computed (or approximated) with linear programming [30], strategy iteration [42] or value iteration [17,1] The maximal total reward in M can be computed using standard techniques such as value iteration and policy iteration [46] as well as the more recent sound value iteration and optimistic value iteration [48,36]. The latter two provide sound precision guarantees for the output value v, i.e., |v − max{Ex M ,s I σ (tot(R * )) | σ ∈ Σ M }| ≤ ε for a given ε > 0.…”