DSMC Evaluation Stages: Fostering Robust and Safe Behavior in Deep Reinforcement Learning

Gros, Timo P.; Höller, Daniel; Hoffmann, Jörg; Klauck, Michaela; Meerkamp, Hendrik; Wolf, Verena

doi:10.1007/978-3-030-85172-9_11

Cited by 6 publications

(14 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from that, we plan to build upon MoGym to develop DSMC techniques further. With DSMC Evaluation Stages [21] it has already been shown that DSMC can be applied during deep RL to determine state space regions with weak performance to concentrate on them during the learning process. With the help of MoGym this technique can now be done much more integrated and there is room for further implementations into this direction in our tool chain.…”

Section: Discussionmentioning

confidence: 99%

“…As such, the environment provides a stable and fully controllable training and checking context to assert the safety risk induced by an agent during and after training. More concrete, MoGym leverages deep statistical model checking (DSMC) [20,21]. As shown in these works on DSMC, the quality assessment of an agent during training is not trivial and can especially not always be derived from the observed training returns.…”

Section: Introductionmentioning

confidence: 99%

“…As shown in these works on DSMC, the quality assessment of an agent during training is not trivial and can especially not always be derived from the observed training returns. Hence, analyzing the quality of the decision-making agents after training clearly is of interest [20,21], especially for badly interpretable agent structures such as neural networks (NN). In DSMC this is done by using the decision-making agent as an oracle resolving the non-determinism in the MDP specifying the environment.…”

Section: Introductionmentioning

confidence: 99%

“…-The DSMC API, also newly implemented on top of Momba. It includes a Python API to use the DSMC functionality [20,21] of the Modest Toolset [13,30]. -DSMC implemented in the Modest Toolset.…”

Section: Introductionmentioning

confidence: 99%

“…-DSMC implemented in the Modest Toolset. In prior work [20,21] We are not aware of any other work that enables a direct connection of formal verification models and reinforcement learning that directly allows the analysis of different RL agents for a variety of verification benchmarks.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

MoGym: Using Formal Models for Training and Verifying Decision-making Agents

Gros

Hermanns

Hoffmann

et al. 2022

Computer Aided Verification

Self Cite

View full text Add to dashboard Cite

MoGym, is an integrated toolbox enabling the training and verification of machine-learned decision-making agents based on formal models, for the purpose of sound use in the real world. Given a formal representation of a decision-making problem in the JANI format and a reach-avoid objective, MoGym (a) enables training a decision-making agent with respect to that objective directly on the model using reinforcement learning (RL) techniques, and (b) it supports rigorous assessment of the quality of the induced decision-making agent by means of deep statistical model checking (DSMC). MoGym implements the standard interface for training environments established by OpenAI Gym, thereby connecting to the vast body of existing work in the RL community. In return, it makes accessible the large set of existing JANI model checking benchmarks to machine learning research. It thereby contributes an efficient feedback mechanism for improving in particular reinforcement learning algorithms. The connective part is implemented on top of Momba. For the DSMC quality assurance of the learned decision-making agents, a variant of the statistical model checker modes of the Modest Toolset is leveraged, which has been extended by two new resolution strategies for non-determinism when encountered during statistical evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MoGym: Using Formal Models for Training and Verifying Decision-making Agents

Gros

Hermanns

Hoffmann

et al. 2022

Computer Aided Verification

Self Cite

View full text Add to dashboard Cite

show abstract

The Modest State of Learning, Sampling, and Verifying Strategies

Hartmanns

Klauck

2022

Leveraging Applications of Formal Methods, Verification and Validation. Adaptation and Learning

Self Cite

View full text Add to dashboard Cite

Optimal decision-making under stochastic uncertainty is a core problem tackled in artificial intelligence/machine learning (AI), planning, and verification. Planning and AI methods aim to find good or optimal strategies to maximise rewards or the probability of reaching a goal. Verification approaches focus on calculating the probability or reward, obtaining the strategy as a side effect. In this paper, we connect three strands of work on obtaining strategies implemented in the context of the Modest Toolset: statistical model checking with either lightweight scheduler sampling or deep learning, and probabilistic model checking. We compare their different goals and abilities, and show newly extended experiments on Racetrack benchmarks that highlight the tradeoffs between the methods. We conclude with an outlook on improving the existing approaches and on generalisations to continuous models, and emphasise the need for further tool development to integrate methods that find, evaluate, compare, and explain strategies.

show abstract

Analyzing neural network behavior through deep statistical model checking

Gros

Hermanns

Hoffmann

et al. 2022

Int J Softw Tools Technol Transfer

Self Cite

View full text Add to dashboard Cite

Neural networks (NN) are taking over ever more decisions thus far taken by humans, even though verifiable system-level guarantees are far out of reach. Neither is the verification technology available, nor is it even understood what a formal, meaningful, extensible, and scalable testbed might look like for such a technology. The present paper is an attempt to improve on both the above aspects. We present a family of formal models that contain basic features of automated decision-making contexts and which can be extended with further orthogonal features, ultimately encompassing the scope of autonomous driving. Due to the possibility to model random noise in the decision actuation, each model instance induces a Markov decision process (MDP) as verification object. The NN in this context has the duty to actuate (near-optimal) decisions. From the verification perspective, the externally learnt NN serves as a determinizer of the MDP, the result being a Markov chain which as such is amenable to statistical model checking. The combination of an MDP and an NN encoding the action policy is central to what we call “deep statistical model checking” (DSMC). While being a straightforward extension of statistical model checking, it enables to gain deep insight into questions like “how high is the NN-induced safety risk?”, “how good is the NN compared to the optimal policy?” (obtained by model checking the MDP), or “does further training improve the NN?”. We report on an implementation of DSMC inside the ModestToolset in combination with externally learnt NNs, demonstrating the potential of DSMC on various instances of the model family, and illustrating its scalability as a function of instance size as well as other factors like the degree of NN training.

show abstract

DSMC Evaluation Stages: Fostering Robust and Safe Behavior in Deep Reinforcement Learning

Cited by 6 publications

References 30 publications

MoGym: Using Formal Models for Training and Verifying Decision-making Agents

MoGym: Using Formal Models for Training and Verifying Decision-making Agents

The Modest State of Learning, Sampling, and Verifying Strategies

Analyzing neural network behavior through deep statistical model checking

Contact Info

Product

Resources

About