MoGym: Using Formal Models for Training and Verifying Decision-making Agents

Abstract

MoGym, is an integrated toolbox enabling the training and verification of machine-learned decision-making agents based on formal models, for the purpose of sound use in the real world. Given a formal representation of a decision-making problem in the JANI format and a reach-avoid objective, MoGym (a) enables training a decision-making agent with respect to that objective directly on the model using reinforcement learning (RL) techniques, and (b) it supports rigorous assessment of the quality of the induced decision-making agent by means of deep statistical model checking (DSMC). MoGym implements the standard interface for training environments established by OpenAI Gym, thereby connecting to the vast body of existing work in the RL community. In return, it makes accessible the large set of existing JANI model checking benchmarks to machine learning research. It thereby contributes an efficient feedback mechanism for improving in particular reinforcement learning algorithms. The connective part is implemented on top of Momba. For the DSMC quality assurance of the learned decision-making agents, a variant of the statistical model checker modes of the Modest Toolset is leveraged, which has been extended by two new resolution strategies for non-determinism when encountered during statistical evaluation.

The Modest State of Learning, Sampling, and Verifying Strategies

“…Deep RL with DSMC Specifics. Usually, learning NN is done on GPUs [43][44][45][46], but for a reasonable runtime comparison, we used a CPU infrastructure here. In addition, the random start setup [44]-during learning, the agent starts randomly from one of the free road cells instead of always from the same start cell-leads to significantly better learning performance.…”

“…The DSMC functionality of using a previously trained NN to resolve the nondeterminism during SMC is implemented in a branch of modes [44] that will be integrated into the official version of the Modest Toolset soon. In addition, this DSMC extension of modes is used in MoGym [45]. MoGym is a toolbox that bridges the gap between formal methods and RL by enabling (a) formally specified training environments to be used with machine-learned decision-making agents, and (b) the rigorous assessment of the quality of learned agents.…”

“…It is used by algorithms interacting with the interface [35,47,68,89,97], as well as benchmarks that implement (and sometimes extend) it [10,27,37,102,103,106]. We use the OpenAI Gym API via MoGym [45] to train NN and then evaluate their quality with DSMC.…”

COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checking

“…This yields the opportunity to verify these formal abstraction and obtaining bounds on the actual policy. [15] is built on top of the MOD-EST toolset [20], and it is related to our tool. The main difference is that our tool supports so-called permissive policies and feature remappings.…”

Analyzing neural network behavior through deep statistical model checking

“…The information delivered by DSMC has already been used to improve reinforcement learning strategies [32] and for the design of policy-analysis tools in synergy with interactive visualization techniques [26,28]. The most important work based on DSMC is MoGym [29], the integrated toolbox enabling the training and verification of machine-learned decisionmaking agents based on formal models, which bridges the reinforcement learning community to formal methods.…”