Verified Probabilistic Policies for Deep Reinforcement Learning

Bacci, Edoardo; Parker, David

doi:10.1007/978-3-031-06773-0_10

Cited by 2 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All experiments were executed on 1 MLflow is a platform to streamline ML development, including tracking experiments, packaging code into reproducible experiments, and sharing and deploying models [41]. 2 We refer the interested reader to the repository https://github.com/LAVA-LAB/ COOL-MC of the tool for more experiments with these and other environments.…”

Section: Numerical Experimentsmentioning

confidence: 99%

“…Thus, the input of COOL-MC consists of two models of the environment: (1) an OpenAI-gym compatible environment, to train an RL policy; (2) an Markov decision process (MDP), specified using the PRISM language [31], to verify the policy together with a formal specification e.g., a probabilistic computation tree logic (PCTL) formula. Only the MDP model of the environment is required: If no OpenAI-gym environment is given, COOL-MC provides a wrapper to cast the MDP as an OpenAI gym environment.…”

Section: Introductionmentioning

confidence: 99%

“…The main difference is that our tool supports so-called permissive policies and feature remappings. Likewise, Bacci et al tackle the problem of verifying stochastic RL policies for continuous environments via an abstraction approach [2] but they abstract the policy using mixed-integer LPs. This limits the NN architectures their tool can handle.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

Gross

Jansen

Junges

et al. 2022

Dependable Software Engineering. Theories, Tools, and Applications

View full text Add to dashboard Cite

This paper presents COOL-MC, a tool that integrates stateof-the-art reinforcement learning (RL) and model checking. Specifically, the tool builds upon the OpenAI gym and the probabilistic model checker Storm. COOL-MC provides the following features: (1) a simulator to train RL policies in the OpenAI gym for Markov decision processes (MDPs) that are defined as input for Storm, (2) a new model builder for Storm, which uses callback functions to verify (neural network) RL policies, (3) formal abstractions that relate models and policies specified in OpenAI gym or Storm, and (4) algorithms to obtain bounds on the performance of so-called permissive policies. We describe the components and architecture of COOL-MC and demonstrate its features on multiple benchmark environments.

show abstract

Section: Numerical Experimentsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

Gross

Jansen

Junges

et al. 2022

Dependable Software Engineering. Theories, Tools, and Applications

View full text Add to dashboard Cite

show abstract

Unifying Qualitative and Quantitative Safety Verification of DNN-Controlled Systems

Zhi,

Wang,

Liu

et al. 2024

Computer Aided Verification

View full text Add to dashboard Cite

The rapid advance of deep reinforcement learning techniques enables the oversight of safety-critical systems through the utilization of Deep Neural Networks (DNNs). This underscores the pressing need to promptly establish certified safety guarantees for such DNN-controlled systems. Most of the existing verification approaches rely on qualitative approaches, predominantly employing reachability analysis. However, qualitative verification proves inadequate for DNN-controlled systems as their behaviors exhibit stochastic tendencies when operating in open and adversarial environments. In this paper, we propose a novel framework for unifying both qualitative and quantitative safety verification problems of DNN-controlled systems. This is achieved by formulating the verification tasks as the synthesis of valid neural barrier certificates (NBCs). Initially, the framework seeks to establish almost-sure safety guarantees through qualitative verification. In cases where qualitative verification fails, our quantitative verification method is invoked, yielding precise lower and upper bounds on probabilistic safety across both infinite and finite time horizons. To facilitate the synthesis of NBCs, we introduce their k-inductive variants. We also devise a simulation-guided approach for training NBCs, aiming to achieve tightness in computing precise certified lower and upper bounds. We prototype our approach into a tool called and showcase its efficacy on four classic DNN-controlled systems.

show abstract

Verified Probabilistic Policies for Deep Reinforcement Learning

Cited by 2 publications

References 32 publications

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

Unifying Qualitative and Quantitative Safety Verification of DNN-Controlled Systems

Contact Info

Product

Resources

About