The Probabilistic Model Checker Storm

Hensel, Christian; Junges, Sebastian; Katoen, Joost-Pieter; Quatmann, Tim; Volk, Matthias

doi:10.48550/arxiv.2002.07080

“…We implemented the computation of abstraction MDPs in C++. Given abstraction MDPs, we compute shields via value iteration with the model checker Storm [26]. All simulations were executed on a desktop computer with a 4 x 2.70 GHz Intel Core i7-7500U CPU, 7.7 GB of RAM running Arch Linux.…”

Section: Implementation and Experimentsmentioning

confidence: 99%

Adaptive Shielding under Uncertainty

Pranger

¹

,

Könighofer

²

,

Tappler

³

et al. 2021

2021 American Control Conference (ACC)

View full text Add to dashboard Cite

This paper targets control problems that exhibit specific safety and performance requirements. In particular, the aim is to ensure that an agent, operating under uncertainty, will at runtime strictly adhere to such requirements. Previous works create so-called shields that correct an existing controller for the agent if it is about to take unbearable safety risks. However, so far, shields do not consider that an environment may not be fully known in advance and may evolve for complex control and learning tasks. We propose a new method for the efficient computation of a shield that is adaptive to a changing environment. In particular, we base our method on problems that are sufficiently captured by potentially infinite Markov decision processes (MDP) and quantitative specifications such as mean payoff objectives. The shield is independent of the controller, which may, for instance, take the form of a high-performing reinforcement learning agent. At runtime, our method builds an internal abstract representation of the MDP and constantly adapts this abstraction and the shield based on observations from the environment. We showcase the applicability of our method via an urban traffic control problem.

show abstract

“…The hyperparameters are β = 1e 3 , β sp = 10, ρ = 1.01, ρ 0 = 1.5, ρ lim = 1e −4 , γ = 0.999, and T = 300. We use the verification tool Storm 1.6.3 [23] and LP solver Gurobi 9.1 [19].…”

Section: Numerical Examplesmentioning

confidence: 99%

Task-Guided Inverse Reinforcement Learning Under Partial Information

Djeumou¹,

Cubuktepe²,

Lennon³

et al. 2021

Preprint

0

View full text Add to dashboard Cite

We study the problem of inverse reinforcement learning (IRL), where the learning agent recovers a reward function using expert demonstrations. Most of the existing IRL techniques make the often unrealistic assumption that the agent has access to full information about the environment. We remove this assumption by developing an algorithm for IRL in partially observable Markov decision processes (POMDPs), where an agent cannot directly observe the current state of the POMDP. The algorithm addresses several limitations of existing techniques that do not take the information asymmetry between the expert and the agent into account. First, it adopts causal entropy as the measure of the likelihood of the expert demonstrations as opposed to entropy in most existing IRL techniques and avoids a common source of algorithmic complexity. Second, it incorporates task specifications expressed in temporal logic into IRL. Such specifications may be interpreted as side information available to the learner a priori in addition to the demonstrations, and may reduce the information asymmetry between the expert and the agent. Nevertheless, the resulting formulation is still nonconvex due to the intrinsic nonconvexity of the so-called forward problem, i.e., computing an optimal policy given a reward function, in POMDPs. We address this nonconvexity through sequential convex programming and introduce several extensions to solve the forward problem in a scalable manner. This scalability allows computing policies that incorporate memory at the expense of added computational cost yet also achieves higher performance compared to memoryless policies. We demonstrate that, even with severely limited data, the algorithm learns reward functions and policies that satisfy the task and induce a similar behavior to the expert by leveraging the side information and incorporating memory into the policy.

show abstract

“…We implemented the computation of abstraction MDPs in C++. Given abstraction MDPs, we compute shields via value iteration with we the model checker Storm [24]. All simulations were executed on a desktop computer with a 4 x 2.70 GHz Intel Core i7-7500U CPU, 7.7 GB of RAM running Arch Linux.…”

Section: Implementation and Experimentsmentioning

confidence: 99%

Adaptive Shielding under Uncertainty

Pranger¹,

Könighofer²,

Tappler³

et al. 2020

Preprint

0

View full text Add to dashboard Cite

This paper targets control problems that exhibit specific safety and performance requirements. In particular, the aim is to ensure that an agent, operating under uncertainty, will at runtime strictly adhere to such requirements. Previous works create so-called shields that correct an existing controller for the agent if it is about to take unbearable safety risks. However, so far, shields do not consider that an environment may not be fully known in advance and may evolve for complex control and learning tasks. We propose a new method for the efficient computation of a shield that is adaptive to a changing environment. In particular, we base our method on problems that are sufficiently captured by potentially infinite Markov decision processes (MDP) and quantitative specifications such as mean payoff objectives. The shield is independent of the controller, which may, for instance, take the form of a high-performing reinforcement learning agent. At runtime, our method builds an internal abstract representation of the MDP and constantly adapts this abstraction and the shield based on observations from the environment. We showcase the applicability of our method via an urban traffic control problem.

show abstract

The Probabilistic Model Checker Storm

Cited by 3 publications

References 81 publications

Adaptive Shielding under Uncertainty

Adaptive Shielding under Uncertainty

Task-Guided Inverse Reinforcement Learning Under Partial Information

Adaptive Shielding under Uncertainty

Contact Info

Product

Resources

About