Monotonic Safety for Scalable and Data-Efficient Probabilistic Safety Analysis

Cleaveland, Matthew; Ruchkin, Ivan; Sokolsky, Oleg; Lee, Insup

doi:10.1109/iccps54341.2022.00015

Cited by 5 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Socha et al [76], in turn, utilized a safe envelope not based on AI to cover wrong braking commands generated by neural networks. Finally, Cleaveland et al [77] identified that their neural network BCS remained formally safe only if specific mathematical properties were held in the target application.…”

Section: Discussionmentioning

confidence: 99%

Design and Assurance of Safety-Critical Systems with Artificial Intelligence in FPGAs: The Safety ArtISt Method and a Case Study of an FPGA-Based Autonomous Vehicle Braking Control System

Silva Neto,

Silva,

Camargo

et al. 2023

Electronics

View full text Add to dashboard Cite

With the advancements in utilizing Artificial Intelligence (AI) in embedded safety-critical systems based on Field-Programmable Gate Arrays (FPGAs), assuring that these systems meet their safety requirements is of paramount importance for their revenue service. Based on this context, this paper has two main objectives. The first of them is to present the Safety ArtISt method, developed by the authors to guide the lifecycle of AI-based safety-critical systems, and emphasize its FPGA-oriented tasks and recommended practice towards safety assurance. The second one is to illustrate the application of Safety ArtISt with an FPGA-based braking control system for autonomous vehicles relying on explainable AI generated with High-Level Synthesis. The results indicate that Safety ArtISt played four main roles in the safety lifecycle of AI-based systems for FPGAs. Firstly, it provided guidance in identifying the safety-critical role of activities such as sensitivity analyses for numeric representation and FPGA dimensioning to achieve safety. Furthermore, it allowed building qualitative and quantitative safety arguments from analyses and physical experimentation with actual FPGAs. It also allowed the early detection of safety issues—thus reducing project costs—and, ultimately, it uncovered relevant challenges not discussed in detail when designing safety-critical, explainable AI for FPGAs.

show abstract

Section: Discussionmentioning

confidence: 99%

Design and Assurance of Safety-Critical Systems with Artificial Intelligence in FPGAs: The Safety ArtISt Method and a Case Study of an FPGA-Based Autonomous Vehicle Braking Control System

Silva Neto,

Silva,

Camargo

et al. 2023

Electronics

View full text Add to dashboard Cite

show abstract

“…Unlike DeepDECS, these approaches assume that the controller has already been synthesized. A number of other approaches [77], [49], [47], [46], [57], [44], [72], [22] have been proposed in recent years for verifying the closed-loop safety of autonomous systems with DNN-based components and already synthesized controllers. These approaches differ in the manner in which they model the environment and perception components but, in general, scalability is a challenge.…”

Section: Related Workmentioning

confidence: 99%

“…As discussed in detail in our related work section, while other approaches that employ deep-learning classifiers for the discrete-event control of AS have been proposed, these approaches focus on the development of end-to-end DNN controllers for AS (e.g. [48], [65]), on quantifying the uncertainty of DNNs to support the probabilistic safety verification of autonomous systems (e.g., [5], [74]), and on verifying the safety of AS with DNN-based components and already implemented controllers (e.g., [22], [44], [46], [47], [49], [57], [72], [77]).…”

Section: Introductionmentioning

confidence: 99%

Controller Synthesis for Autonomous Systems With Deep-Learning Perception Components

Calinescu,

Imrie,

Mangal

et al. 2024

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

We present DeepDECS, a new method for the synthesis of correct-by-construction software controllers for autonomous systems that use deep neural network (DNN) classifiers for the perception step of their decisionmaking processes. Despite major advances in deep learning in recent years, providing safety guarantees for these systems remains very challenging. Our controller synthesis method addresses this challenge by integrating DNN verification with the synthesis of verified Markov models. The synthesised models correspond to discreteevent software controllers guaranteed to satisfy the safety, dependability and performance requirements of the autonomous system, and to be Pareto optimal with respect to a set of optimisation objectives. We evaluate the method in simulation by using it to synthesise controllers for mobile-robot collision limitation, and for maintaining driver attentiveness in shared-control autonomous driving.

show abstract

“…Second, it is often unknown in which direction the probabilities need to be shifted to induce a conservative shift to the model. One opportunity is to use monotonic safety rules [11]; for now, this remains a promising and important future research direction.…”

Section: Conservatism Guaranteesmentioning

confidence: 99%

Conservative Safety Monitors of Stochastic Dynamical Systems

Cleaveland¹,

Ruchkin²,

Sokolsky³

et al. 2023

Preprint

View full text Add to dashboard Cite

Generating accurate runtime safety estimates for autonomous systems is vital to ensuring their continued proliferation. However, accurately reasoning about future system behaviors is generally too complex to do at runtime. To better reason about system safety at runtime, we propose a method for leveraging design time model checking results at runtime. Specifically, we model the system as a probabilistic automaton (PA) and compute bounded time reachability probabilities over the states of the PA at design time. At runtime, we combine distributions of state estimates with the safety probabilities from design time to produce a bounded time safety estimate. We argue that our approach produces well calibrated safety probabilities, assuming the estimated state distributions are well calibrated. We evaluate our approach using a case study of simulated water tanks.

show abstract

Monotonic Safety for Scalable and Data-Efficient Probabilistic Safety Analysis

Cited by 5 publications

References 21 publications

Design and Assurance of Safety-Critical Systems with Artificial Intelligence in FPGAs: The Safety ArtISt Method and a Case Study of an FPGA-Based Autonomous Vehicle Braking Control System

Design and Assurance of Safety-Critical Systems with Artificial Intelligence in FPGAs: The Safety ArtISt Method and a Case Study of an FPGA-Based Autonomous Vehicle Braking Control System

Controller Synthesis for Autonomous Systems With Deep-Learning Perception Components

Conservative Safety Monitors of Stochastic Dynamical Systems

Contact Info

Product

Resources

About