Parameter Synthesis in Markov Models: A Gentle Survey

Jansen, Nils; Junges, Sebastian; Katoen, Joost-Pieter

doi:10.1007/978-3-031-22337-2_20

Cited by 9 publications

(4 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The key of our approach to tackle various synthesis and inference problems on (p)BNs is to exploit model-checking techniques on MCs (Baier & Katoen, 2008;Katoen, 2016;Baier et al, 2018) and synthesis techniques (Junges et al, 2019) on pMCs. To that end, we transform a (p)BN into a (p)MC.…”

Section: From Bayesian Network To Markov Chainsmentioning

confidence: 99%

“…a given objective, e.g., is the probability to reach some states below (or above) a given threshold λ? Due to algorithmic improvements, nowadays Markov models with hundreds of thousands of states and tens or hundreds of parameters are in reach (Dehnert et al, 2015;Quatmann et al, 2016;Gainer et al, 2018;Fang et al, 2021;Heck et al, 2022); for a recent overview see (Junges et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Automatically Finding the Right Probabilities in Bayesian Networks

Salmani,

Katoen

2023

jair

View full text Add to dashboard Cite

This paper presents alternative techniques for inference on classical Bayesian networks in which all probabilities are fixed, and for synthesis problems when conditional probability tables (CPTs) in such networks contain symbolic parameters rather than concrete probabilities. The key idea is to exploit probabilistic model checking as well as its recent extension to parameter synthesis techniques for parametric Markov chains. To enable this, the Bayesian networks are transformed into Markov chains and their objectives are mapped onto probabilistic temporal logic formulas. For exact inference, we compare probabilistic model checking to weighted model counting on various Bayesian network benchmarks. We contrast symbolic model checking using multi-terminal binary (aka: algebraic) decision diagrams to symbolic inference using proba- bilistic sentential decision diagrams, symbolic data structures that are tailored to Bayesian networks. For the parametric setting, we describe how our techniques can be used for various synthesis problems such as computing sensitivity functions (and values), simple and difference parameter tuning and ratio parameter tuning. Our parameter synthesis techniques are applicable to arbitrarily many, possibly dependent, parameters that may occur in multiple CPTs. This lifts restrictions, e.g., on the number of parametrized CPTs, or on parameter dependencies between several CPTs, that exist in the literature. Experiments on several benchmarks show that our parameter synthesis techniques can treat parameter synthesis for Bayesian networks (with hundreds of unknown parameters) that are out of reach for existing techniques.

show abstract

Section: From Bayesian Network To Markov Chainsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Automatically Finding the Right Probabilities in Bayesian Networks

Salmani,

Katoen

2023

jair

View full text Add to dashboard Cite

show abstract

“…Interval MDPs [25,43,23] and SGs [38] do not allow for dependencies between states and thus cannot model features such as various obstacle positions. Parametric MDPs [2,44,24] assume controllable uncertainty and do not consider robustness of policies.…”

Section: Related Workmentioning

confidence: 99%

Robust Almost-Sure Reachability in Multi-Environment MDPs

Vegt¹,

Jansen²,

Junges³

2023

Preprint

View full text Add to dashboard Cite

Multiple-environment MDPs (MEMDPs) capture finite sets of MDPs that share the states but differ in the transition dynamics. These models form a proper subclass of partially observable MDPs (POMDPs). We consider the synthesis of policies that robustly satisfy an almost-sure reachability property in MEMDPs, that is, one policy that satisfies a property for all environments. For POMDPs, deciding the existence of robust policies is an EXPTIME-complete problem. In this paper, we show that this problem is PSPACE-complete for MEMDPs, while the policies in general require exponential memory. We exploit the theoretical results to develop and implement an algorithm that shows promising results in synthesizing robust policies for various benchmarks.1 Hidden-parameter MDPs are different than MEMDPs in that they assume a prior over different MDPs. However, for almost-sure properties, this difference is irrelevant.

show abstract

“…[42] employs a game-based abstraction approach to efficiently solve problems with specific properties. In [22], finite-state controllers for POMDPs are computed using parameter synthesis for Markov chains [19,21] by applying convex optimization techniques [12,13]. Another work employs machine learning techniques together with formal verification to achieve sound but not optimal solutions [7].…”

Section: Docked In Left Stationmentioning

confidence: 99%

Strengthening Deterministic Policies for POMDPs

Winterer

Wimmer

Jansen

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The synthesis problem for partially observable Markov decision processes (POMDPs) is to compute a policy that satisfies a given specification. Such policies have to take the full execution history of a POMDP into account, rendering the problem undecidable in general. A common approach is to use a limited amount of memory and randomize over potential choices. Yet, this problem is still NP-hard and often computationally intractable in practice. A restricted problem is to use neither history nor randomization, yielding policies that are called stationary and deterministic. Previous approaches to compute such policies employ mixed-integer linear programming (MILP). We provide a novel MILP encoding that supports sophisticated specifications in the form of temporal logic constraints. It is able to handle an arbitrary number of such specifications. Yet, randomization and memory are often mandatory to achieve satisfactory policies. First, we extend our encoding to deliver a restricted class of randomized policies. Second, based on the results of the original MILP, we employ a preprocessing of the POMDP to encompass memory-based decisions. The advantages of our approach over state-of-the-art POMDP solvers lie (1) in the flexibility to strengthen simple deterministic policies without losing computational tractability and (2) in the ability to enforce the provable satisfaction of arbitrarily many specifications. The latter point allows to take trade-offs between performance and safety aspects of typical POMDP examples into account. We show the effectiveness of our method on a broad range of benchmarks.

show abstract

Parameter Synthesis in Markov Models: A Gentle Survey

Cited by 9 publications

References 78 publications

Automatically Finding the Right Probabilities in Bayesian Networks

Automatically Finding the Right Probabilities in Bayesian Networks

Robust Almost-Sure Reachability in Multi-Environment MDPs

Strengthening Deterministic Policies for POMDPs

Contact Info

Product

Resources

About