Multi-objective Optimization of Long-run Average and Total Rewards

Abstract

This paper presents an efficient procedure for multi-objective model checking of long-run average reward (aka: mean pay-off) and total reward objectives as well as their combination. We consider this for Markov automata, a compositional model that captures both traditional Markov decision processes (MDPs) as well as a continuous-time variant thereof. The crux of our procedure is a generalization of Forejt et al.’s approach for total rewards on MDPs to arbitrary combinations of long-run and total reward objectives on Markov automata. Experiments with a prototypical implementation on top of the Storm model checker show encouraging results for both model types and indicate a substantial improved performance over existing multi-objective long-run MDP model checking based on linear programming.

Pareto Curves for Compositionally Model Checking String Diagrams of MDPs

“…For an oMDP A, we can efficiently approximate Ach A via an off-the-shelf multiobjective model checking algorithm [12,30]. We outline this algorithm, tailored to oMDPs.…”

“…L and U are initialized as a trivial approximation. The algorithm iteratively refines them by computing weighted reachability probabilities for some weight vector w ∈ W O , which is adequately chosen [30] 7 . For that w, we denote the weighted reachability using WRPr A,σ (w, i) := j∈O w j • RPr A,σ (i, j).…”

“…As the context of the open MDP determines the preferred trade-off, we compute all schedulers that are optimal for a specific trade-off. These schedulers are Pareto-optimal and their computation is well-studied [9,12,30].…”

Multi-objective Optimization of Long-run Average and Total Rewards

“…The implementation, models, and log files are available at [49].…”