Runtime Monitors for Markov Decision Processes

Junges, Sebastian; Torfah, Hazem; Seshia, Sanjit A.

doi:10.1007/978-3-030-81688-9_26

Cited by 13 publications

(12 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To understand the practical performance of the different algorithms, we performed an extensive experimental evaluation. We used three sets of benchmarks: all applicable benchmark instances 18 from the Quantitative Verification Benchmark Set (QVBS) [30] (the qvbs set), a subset of hard QVBS instances (the hard set), and numerically challenging models from a runtime monitoring application [32] (the premise set, named for the corresponding prototype). We consider two probabilistic model checkers, Storm [31] and the Modest Toolset's [28] mcsta.…”

Section: Experimental Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

A Practitioner's Guide to MDP Model Checking Algorithms

Hartmanns¹,

Junges²,

Quatmann³

et al. 2023

Preprint

View full text Add to dashboard Cite

Model checking undiscounted reachability and expected-reward properties on Markov decision processes (MDPs) is key for the verification of systems that act under uncertainty. Popular algorithms are policy iteration and variants of value iteration; in tool competitions, most participants rely on the latter. These algorithms generally need worst-case exponential time. However the problem can equally be formulated as a linear program, solvable in polynomial time. In this paper, we give a detailed overview of today's state-of-the-art algorithms for MDP model checking with a focus on performance and correctness. We highlight their fundamental differences, and describe various optimisations and implementation variants. We experimentally compare floating-point and exact-arithmetic implementations of all algorithms on three benchmark sets using two probabilistic model checkers. Our results show that (optimistic) value iteration is a sensible default, but other algorithms are preferable in specific settings. This paper thereby provides a guide for MDP verification practitioners-tool builders and users alike.

show abstract

Section: Experimental Evaluationmentioning

confidence: 99%

“…Indeed, for stochastic games and a particular technique, using LP to solve the underlying MDPs may be preferential [3,Appendix E.4]. For examples in runtime assurance, numerical instability meant that PI was preferred [32,Sect. 6].…”

Section: Introductionmentioning

confidence: 99%

A Practitioner's Guide to MDP Model Checking Algorithms

Hartmanns¹,

Junges²,

Quatmann³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Junges et al [39] present a runtime monitoring approach for partially observable systems with non-deterministic and probabilistic dynamics. The approach is based on traces of observations on models that combine non-determinism and probabilities.…”

Section: Approaches For Efficient Verificationmentioning

confidence: 99%

Deep Learning for Effective and Efficient Reduction of Large Adaptation Spaces in Self-Adaptive Systems

Weyns¹,

Gheibi²,

Quin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Many software systems today face uncertain operating conditions, such as sudden changes in the availability of resources or unexpected user behavior. Without proper mitigation these uncertainties can jeopardize the system goals. Self-adaptation is a common approach to tackle such uncertainties. When the system goals may be compromised, the self-adaptive system has to select the best adaptation option to reconfigure by analyzing the possible adaptation options, i.e., the adaptation space. Yet, analyzing large adaptation spaces using rigorous methods can be resource-and time-consuming, or even be infeasible. One approach to tackle this problem is by using online machine learning to reduce adaptation spaces. However, existing approaches require domain expertise to perform feature engineering to define the learner, and support online adaptation space reduction only for specific goals. To tackle these limitations, we present "Deep Learning for Adaptation Space Reduction Plus" -DLASeR+ in short. DLASeR+ offers an extendable learning framework for online adaptation space reduction that does not require feature engineering, while supporting three common types of adaptation goals: threshold, optimization, and set-point goals. We evaluate DLASeR+ on two instances of an Internet-of-Things application with increasing sizes of adaptation spaces for different combinations of adaptation goals. We compare DLASeR+ with a baseline that applies exhaustive analysis and two stateof-the-art approaches for adaptation space reduction that rely on learning. Results show that DLASeR+ is effective with a negligible effect on the realization of the adaptation goals compared to an exhaustive analysis approach, and supports three common types of adaptation goals beyond the state-of-the-art approaches.CCS Concepts: • Software and its engineering → Software design engineering; • Theory of computation → Online learning algorithms.

show abstract

Section: Approaches For Eficient Verificationmentioning

confidence: 99%

Deep Learning for Effective and Efficient Reduction of Large Adaptation Spaces in Self-adaptive Systems

Weyns

Gheibi

Quin

et al. 2022

ACM Trans. Auton. Adapt. Syst.

View full text Add to dashboard Cite

Many software systems today face uncertain operating conditions, such as sudden changes in the availability of resources or unexpected user behavior. Without proper mitigation these uncertainties can jeopardize the system goals. Self-adaptation is a common approach to tackle such uncertainties. When the system goals may be compromised, the self-adaptive system has to select the best adaptation option to reconfigure by analyzing the possible adaptation options, i.e., the adaptation space. Yet, analyzing large adaptation spaces using rigorous methods can be resource- and time-consuming, or even be infeasible. One approach to tackle this problem is by using online machine learning to reduce adaptation spaces. However, existing approaches require domain expertise to perform feature engineering to define the learner, and support online adaptation space reduction only for specific goals. To tackle these limitations, we present ”Deep Learning for Adaptation Space Reduction Plus” – DLASeR+ in short. DLASeR+ offers an extendable learning framework for online adaptation space reduction that does not require feature engineering, while supporting three common types of adaptation goals: threshold, optimization, and set-point goals. We evaluate DLASeR+ on two instances of an Internet-of-Things application with increasing sizes of adaptation spaces for different combinations of adaptation goals. We compare DLASeR+ with a baseline that applies exhaustive analysis and two state-of-the-art approaches for adaptation space reduction that rely on learning. Results show that DLASeR+ is effective with a negligible effect on the realization of the adaptation goals compared to an exhaustive analysis approach, and supports three common types of adaptation goals beyond the state-of-the-art approaches.

show abstract

Runtime Monitors for Markov Decision Processes

Cited by 13 publications

References 55 publications

A Practitioner's Guide to MDP Model Checking Algorithms

A Practitioner's Guide to MDP Model Checking Algorithms

Deep Learning for Effective and Efficient Reduction of Large Adaptation Spaces in Self-Adaptive Systems

Deep Learning for Effective and Efficient Reduction of Large Adaptation Spaces in Self-adaptive Systems

Contact Info

Product

Resources

About