Shield Synthesis for Real: Enforcing Safety in Cyber-Physical Systems

Wu, Mingquan; Wang, Jingbo; Deshmukh, Jyotirmoy V.; Wang, Chao

doi:10.23919/fmcad.2019.8894264

Cited by 21 publications

(18 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Shields are usually constructed offline by computing a maximally permissive policy containing all actions that will not violate the safety specification. Several extensions exist [4,6,29,39]. The shielding approach has been shown to be successful in combination with RL [2,21].…”

Section: Related Workmentioning

confidence: 99%

Online Shielding for Stochastic Systems

Könighofer

Rudolf

Palmisano

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We propose a method to develop trustworthy reinforcement learning systems. To ensure safety especially during exploration, we automatically synthesize a correct-by-construction runtime enforcer, called a shield, that blocks all actions of the agent that are unsafe with respect to a temporal logic specification. Our main contribution is a new synthesis algorithm for computing the shield online. Existing offline shielding approaches compute exhaustively the safety of all states-action combinations ahead-of-time, resulting in huge computation times, large memory consumption, and significant delays at runtime due to the look-ups in huge databases. The intuition behind online shielding is to compute at runtime the set of all states that could be reached in the near future. For each of these states, the safety of all available actions is analysed and used for shielding as soon as one of the considered states is reached. Our proposed method is general and can be applied to a wide range of planning problems with stochastic behaviour. For our evaluation, we selected a 2player version of the classical computer game Snake. The game requires fast decisions and the multiplayer setting induces a large state space, computationally expensive to analyze exhaustively. The safety objective of collision avoidance is easily transferable to a variety of planning tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Online Shielding for Stochastic Systems

Könighofer

Rudolf

Palmisano

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Several formally based runtime enforcement approaches have been proposed [16]- [19], which are not suitable for autonomous systems as they are reactive in nature. Our work is related to the class of runtime enforcement techniques that are suitable for reactive systems [12], [20], [21]. These rely on low-overhead wrappers, which mediate between the environment and the controller of a reactive system, to ensure that the system operates safely at all times by ensuring that all user specified policies hold.…”

Section: Related Workmentioning

confidence: 99%

Runtime Interchange for Adaptive Re-use of Intelligent Cyber-Physical System Controllers

Pearce,

Yang,

Pinisetty

et al. 2021

Preprint

View full text Add to dashboard Cite

Cyber-Physical Systems (CPSs) such as those found within autonomous vehicles are increasingly adopting Artificial Neural Network (ANN)-based controllers. To ensure the safety of these controllers, there is a spate of recent activity to formally verify the ANN-based designs. There are two challenges with these approaches: (1) The verification of such systems is difficult and time consuming. (2) These verified controllers are not able to adapt to frequent requirements changes, which are typical in situations like autonomous driving. This raises the question: how can trained and verified controllers, which have gone through expensive training and verification processes, be re-used to deal with requirement changes?This paper addresses this challenge for the first time by proposing a new framework that is capable of dealing with requirement changes at runtime through a mechanism we term runtime interchange. Our approach functions via a continual exchange and selection process of multiple pre-verified controllers. It represents a key step on the way to componentoriented engineering for intelligent designs, as it preserves the behaviours of the original controllers while introducing additional functionality. To demonstrate the efficacy of our approach we utilise an existing autonomous driving case study as well as a set of smaller benchmarks. These show that introduced overheads are extremely minimal and that the approach is very scalable.

show abstract

“…Run-time Monitoring and Anomaly Detection in CPS: Recent works on run-time safety monitoring in CPS focus on control invariant methods [86], dynamic invariant detection [20], application-dependent multi-level monitoring [87], unsupervised anomaly detection from streaming data [88], [89], and run-time safety guards that satisfy a predefined set of safety properties [90], [91].…”

Section: Related Workmentioning

confidence: 99%

Data-driven Design of Context-aware Monitors for Hazard Prediction in Artificial Pancreas Systems

Xugui¹,

Ahmed²,

Aylor³

et al. 2021

Preprint

View full text Add to dashboard Cite

Medical Cyber-physical Systems (MCPS) are vulnerable to accidental or malicious faults that can target their controllers and cause safety hazards and harm to patients. This paper proposes a combined model and data-driven approach for designing context-aware monitors that can detect early signs of hazards and mitigate them in MCPS. We present a framework for formal specification of unsafe system context using Signal Temporal Logic (STL) combined with an optimization method for patient-specific refinement of STL formulas based on real or simulated faulty data from the closed-loop system for the generation of monitor logic. We evaluate our approach in simulation using two state-of-the-art closed-loop Artificial Pancreas Systems (APS). The results show the context-aware monitor achieves up to 1.4 times increase in average hazard prediction accuracy (F1score) over several baseline monitors, reduces false-positive and false-negative rates, and enables hazard mitigation with a 54% success rate while decreasing the average risk for patients.

show abstract

Shield Synthesis for Real: Enforcing Safety in Cyber-Physical Systems

Cited by 21 publications

References 42 publications

Online Shielding for Stochastic Systems

Online Shielding for Stochastic Systems

Runtime Interchange for Adaptive Re-use of Intelligent Cyber-Physical System Controllers

Data-driven Design of Context-aware Monitors for Hazard Prediction in Artificial Pancreas Systems

Contact Info

Product

Resources

About