Assurance in Reinforcement Learning Using Quantitative Verification

Călinescu, Radu; Kudenko, Daniel; Banks, Alec

doi:10.1007/978-3-319-66790-4_5

Cited by 11 publications

(14 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This area of future work was made possible by the recent adoption of our approach within several projects carried out by teams that include researchers and engineers not involved in the EvoChecker development. These projects have used or will use EvoChecker to devise safe reinforcement learning solutions (Mason et al 2017(Mason et al , 2018, to synthesise robust designs for software-based systems (Calinescu et al 2017b, c), and to suggest safe evacuation routes for communities affected by adverse events such as natural disasters. This will show how easy it is to define and validate EvoChecker models and requirements in real applications, allowing us to improve the usability of the approach.…”

Section: Discussionmentioning

confidence: 99%

Synthesis of probabilistic models for quality-of-service software engineering

Gerasimou

Călinescu

Tamburrelli³

2018

Autom Softw Eng

Self Cite

View full text Add to dashboard Cite

An increasingly used method for the engineering of software systems with strict quality-of-service (QoS) requirements involves the synthesis and verification of probabilistic models for many alternative architectures and instantiations of system parameters. Using manual trial-and-error or simple heuristics for this task often produces suboptimal models, while the exhaustive synthesis of all possible models is typically intractable. The EvoChecker search-based software engineering approach presented in our paper addresses these limitations by employing evolutionary algorithms to automate the model synthesis process and to significantly improve its outcome. EvoChecker can be used to synthesise the Pareto-optimal set of probabilistic models associated with the QoS requirements of a system under design, and to support the selection of a suitable system architecture and configuration. EvoChecker can also be used at runtime, to drive the efficient reconfiguration of a self-adaptive software system. We evaluate EvoChecker on several variants of three systems from different application domains, and show its effectiveness and applicability. Keywords Search-based software engineering • Probabilistic model checking • Evolutionary algorithms • QoS requirements B Simos Gerasimou

show abstract

Section: Discussionmentioning

confidence: 99%

Synthesis of probabilistic models for quality-of-service software engineering

Gerasimou

Călinescu

Tamburrelli³

2018

Autom Softw Eng

Self Cite

View full text Add to dashboard Cite

show abstract

“…In recent work, we used Markov decision processes (MDPs) to model an assisted-living SCPS developed to help dementia sufferers with the daily task of hand-washing [12]. The SCPS provided voice prompts to the sufferers in certain MDP states, to guide them through what they must do next, if they were struggling to progress.…”

Section: Examplementioning

confidence: 99%

“…Example 3 Consider again the route-planning and assistedliving SCPS from Examples 1 and 2. While probabilistic temporal logics were successfully used to specify requirements associated with the risks and duration of evacuation routes [1] and with the sequence of voice prompts provided to dementia sufferers [12], these logics cannot easily express requirements such as the interactions between evacuees who use the same route, or the distress experienced by sufferers who receive too many reminders or do not see their carers for long periods of time (open challenge OC1). Furthermore, the effectiveness of these SCPS depends on the accuracy with which events (e.g., damage to the road infrastructure) in the evacuated area and sufferer response to voice prompts, respectively, are mapped to state transition probabilities within the stochastic models that underpin decision making in these systems (OC2).…”

Section: Oc2) Ensuring the Accuracy Of Stochastic Models Of Scps-mentioning

confidence: 99%

Socio-Cyber-Physical Systems: Models, Opportunities, Open Challenges

Călinescu¹,

Cámara

Paterson³

2019

2019 IEEE/ACM 5th International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS)

Self Cite

View full text Add to dashboard Cite

Almost without exception, cyber-physical systems operate alongside, for the benefit of, and supported by humans. Unsurprisingly, disregarding their social aspects during development and operation renders these systems ineffective. In this paper, we explore approaches to modelling and reasoning about the human involvement in socio-cyber-physical systems (SCPS). To provide an unbiased perspective, we describe both the opportunities afforded by the presence of human agents, and the challenges associated with ensuring that their modelling is sufficiently accurate to support decision making during SCPS development and, if applicable, at run-time. Using SCPS examples from emergency management and assisted living, we illustrate how recent advances in stochastic modelling, analysis and synthesis can be used to exploit human observations about the impact of natural and man-made disasters, and to support the efficient provision of assistive care.

show abstract

“…To alleviate this problem, (D)RL algorithms are being combined with formal verification techniques to ensure safety in learning. Even though significant progress has been achieved in this direction [1,5,9,12,19,22], settings with multiple learning agents are comparatively less explored and understood.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper we introduce assured multi-agent reinforcement learning (AMARL), a method to formally guarantee the safe behaviour of agents acting in an unknown environment through the satisfaction of safety constraints by the solution learned using a DRL algorithm, both at training and test time. Building upon the assured reinforcement learning (ARL) technique in [19], we combine reinforcement learning and formal verification [13] to ensure the satisfaction of constraints expressed in Probabilistic Computation Tree Logic (PCTL) [11]. Differently from ARL, we support a multi-agent setting and DRL algorithms.…”

Section: Introductionmentioning

confidence: 99%

An Abstraction-based Method to Check Multi-Agent Deep Reinforcement-Learning Behaviors

Mqirmi¹,

Belardinelli²,

León³

2021

Preprint

View full text Add to dashboard Cite

Multi-agent reinforcement learning (RL) often struggles to ensure the safe behaviours of the learning agents, and therefore it is generally not adapted to safety-critical applications. To address this issue, we present a methodology that combines formal verification with (deep) RL algorithms to guarantee the satisfaction of formally-specified safety constraints both in training and testing. The approach we propose expresses the constraints to verify in Probabilistic Computation Tree Logic (PCTL) and builds an abstract representation of the system to reduce the complexity of the verification step. This abstract model allows for model checking techniques to identify a set of abstract policies that meet the safety constraints expressed in PCTL. Then, the agents' behaviours are restricted according to these safe abstract policies. We provide formal guarantees that by using this method, the actions of the agents always meet the safety constraints, and provide a procedure to generate an abstract model automatically. We empirically evaluate and show the effectiveness of our method in a multi-agent environment.

show abstract

Assurance in Reinforcement Learning Using Quantitative Verification

Cited by 11 publications

References 36 publications

Synthesis of probabilistic models for quality-of-service software engineering

Synthesis of probabilistic models for quality-of-service software engineering

Socio-Cyber-Physical Systems: Models, Opportunities, Open Challenges

An Abstraction-based Method to Check Multi-Agent Deep Reinforcement-Learning Behaviors

Contact Info

Product

Resources

About