Verification of Indefinite-Horizon POMDPs

Bork, Alexander; Junges, Sebastian; Katoen, Joost-Pieter; Quatmann, Tim

doi:10.1007/978-3-030-59152-6_16

Cited by 23 publications

(41 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…State beliefs are studied when verifying HMMs [59], where the question whether a sequence of observations likely occurs, or which HMM is an adequate representation of a system [37]. State beliefs are prominent in the verification of partially observable MDPs [16,32,40], where one can observe the actions taken (but the problem itself is to find the right scheduler). Our monitoring problem can be phrased as a special case of verification of partially observable stochastic games [20], but automatic techniques for those very general models are lacking.…”

Section: Related Workmentioning

confidence: 99%

Runtime Monitors for Markov Decision Processes

Junges

Torfah

Seshia

2021

Computer Aided Verification

Self Cite

View full text Add to dashboard Cite

We investigate the problem of monitoring partially observable systems with nondeterministic and probabilistic dynamics. In such systems, every state may be associated with a risk, e.g., the probability of an imminent crash. During runtime, we obtain partial information about the system state in form of observations. The monitor uses this information to estimate the risk of the (unobservable) current system state. Our results are threefold. First, we show that extensions of state estimation approaches do not scale due the combination of nondeterminism and probabilities. While exploiting a geometric interpretation of the state estimates improves the practical runtime, this cannot prevent an exponential memory blowup. Second, we present a tractable algorithm based on model checking conditional reachability probabilities. Third, we provide prototypical implementations and manifest the applicability of our algorithms to a range of benchmarks. The results highlight the possibilities and boundaries of our novel algorithms.

show abstract

Section: Related Workmentioning

confidence: 99%

Runtime Monitors for Markov Decision Processes

Junges

Torfah

Seshia

2021

Computer Aided Verification

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our experiments indicate that this approach clearly outperforms existing algorithms based on linear programming. Future work includes lifting the approach to partially observable MDP and stochastic games, potentially using ideas of [10] and [2], respectively.…”

Section: Resultsmentioning

confidence: 99%

Multi-objective Optimization of Long-run Average and Total Rewards

Quatmann

Katoen

2021

Tools and Algorithms for the Construction and Analysis of Systems

Self Cite

View full text Add to dashboard Cite

This paper presents an efficient procedure for multi-objective model checking of long-run average reward (aka: mean pay-off) and total reward objectives as well as their combination. We consider this for Markov automata, a compositional model that captures both traditional Markov decision processes (MDPs) as well as a continuous-time variant thereof. The crux of our procedure is a generalization of Forejt et al.’s approach for total rewards on MDPs to arbitrary combinations of long-run and total reward objectives on Markov automata. Experiments with a prototypical implementation on top of the Storm model checker show encouraging results for both model types and indicate a substantial improved performance over existing multi-objective long-run MDP model checking based on linear programming.

show abstract

“…Such a belief MDP has additional properties that have been exploited to allow verification [80,98,102]. Storm uses a combination of abstraction-and-refinement techniques to iteratively generate a finite abstract belief MDP that soundly approximates the extremal reachability probabilities in the POMDP [19].…”

Section: Partially Observable Markov Decision Processesmentioning

confidence: 99%

The probabilistic model checker Storm

Hensel

Junges

Katoen

et al. 2021

Int J Softw Tools Technol Transfer

Self Cite

108

View full text Add to dashboard Cite

We present the probabilistic model checker Storm. Storm supports the analysis of discrete- and continuous-time variants of both Markov chains and Markov decision processes. Storm has three major distinguishing features. It supports multiple input languages for Markov models, including the Jani and Prism modeling languages, dynamic fault trees, generalized stochastic Petri nets, and the probabilistic guarded command language. It has a modular setup in which solvers and symbolic engines can easily be exchanged. Its Python API allows for rapid prototyping by encapsulating Storm’s fast and scalable algorithms. This paper reports on the main features of Storm and explains how to effectively use them. A description is provided of the main distinguishing functionalities of Storm. Finally, an empirical evaluation of different configurations of Storm on the QComp 2019 benchmark set is presented.

show abstract

Verification of Indefinite-Horizon POMDPs

Cited by 23 publications

References 25 publications

Runtime Monitors for Markov Decision Processes

Runtime Monitors for Markov Decision Processes

Multi-objective Optimization of Long-run Average and Total Rewards

The probabilistic model checker Storm

Contact Info

Product

Resources

About