Reasoning about safety of learning-enabled components in autonomous cyber-physical systems

Tuncali, Cumhur Erkan; Ito, Hideto; Kapinski, James; Deshmukh, Jyotirmoy V.

doi:10.1145/3195970.3199852

Cited by 41 publications

(14 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tuncali et al [164] use barrier certificates for proof of safety. According to [156], they are similar to Lyapunov functions, but focus on safety instead of stability.…”

Section: Reachability Analysismentioning

confidence: 99%

See 1 more Smart Citation

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

When will automated vehicles come onto the market? This question has puzzled the automotive industry and society for years. The technology and its implementation have made rapid progress over the last decade, but the challenge of how to prove the safety of these systems has not yet been solved. Since a market launch without proof of safety would neither be accepted by society nor by legislators, much time and many resources have been invested into safety assessment in recent years in order to develop new approaches for an efficient assessment. This paper therefore provides an overview of various approaches, and gives a comprehensive survey of the so-called scenario-based approach. The scenario-based approach is a promising method, in which individual traffic situations are typically tested by means of virtual simulation. Since an infinite number of different scenarios can theoretically occur in real-world traffic, even the scenario-based approach leaves the question unanswered as to how to break these down into a finite set of scenarios, and find those which are representative in order to render testing more manageable. This paper provides a comprehensive literature review of related safety-assessment publications that deal precisely with this question. Therefore, this paper develops a novel taxonomy for the scenario-based approach, and classifies all literature sources. Based on this, the existing methods will be compared with each other and, as one conclusion, the alternative concept of formal verification will be combined with the scenario-based approach. Finally, future research priorities are derived.

show abstract

“…Tuncali et al [164] use barrier certificates for proof of safety. According to [156], they are similar to Lyapunov functions, but focus on safety instead of stability.…”

Section: Reachability Analysismentioning

confidence: 99%

“…Most verification approaches focus on the planning module. [164] can be seen as a starting point for verifying the perception system with barrier certificates. Even in scenario-based testing, there are not many publications that focus particularly on the evaluation of perception [171], [172].…”

Section: ) Functional Decomposition For Scenario Reductionmentioning

confidence: 99%

Survey on Scenario-Based Safety Assessment of Automated Vehicles

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Instead of computing reachable sets, a different approach for verifying neural network controlled systems is through barrier certificate synthesis. Tuncali et al synthesize candidate barrier certificates using simulation-guided techniques, and then verify the overall system safety by checking the validity of the barrier certificate conditions for the candidate [35]. The safety property was proofed, or a counterexample was returned to updated candidate barrier certificates.…”

Section: Related Workmentioning

confidence: 99%

An Iterative Scheme of Safe Reinforcement Learning for Nonlinear Systems via Barrier Certificate Generation

Yang

Zhang

Wang

et al. 2021

Computer Aided Verification

View full text Add to dashboard Cite

In this paper, we propose a safe reinforcement learning approach to synthesize deep neural network (DNN) controllers for nonlinear systems subject to safety constraints. The proposed approach employs an iterative scheme where a learner and a verifier interact to synthesize safe DNN controllers. The learner trains a DNN controller via deep reinforcement learning, and the verifier certifies the learned controller through computing a maximal safe initial region and its corresponding barrier certificate, based on polynomial abstraction and bilinear matrix inequalities solving. Compared with the existing verification-in-the-loop synthesis methods, our iterative framework is a sequential synthesis scheme of controllers and barrier certificates, which can learn safe controllers with adaptive barrier certificates rather than user-defined ones. We implement the tool SRLBC and evaluate its performance over a set of benchmark examples. The experimental results demonstrate that our approach efficiently synthesizes safe DNN controllers even for a nonlinear system with dimension up to 12.

show abstract

“…However, the vast majority of these techniques have only been able to deal with feed-forward neural networks with piecewise-linear activation functions [4]. Additionally, the bulk of these methods have primarily considered the verification of input-output properties of neural networks in isolation [22], and there are only a handful of works that have explicitly addressed the verification of closed-loop control systems with neural network controllers [5,8,[19][20][21]. One of the central challenges in verifying neural network control systems is that applying existing methodology to these systems is not straightforward [9], and a simple combination of verification tools for non-linear ordinary differential equations along with a neural network reachability tool suffers from severe overestimation errors [5].…”

Section: Context and Originsmentioning

confidence: 99%

“…In this case, we have constraints on the inputs which can only take on two possible values each: θ, φ ∈ {−0. [20,20,8] neurons respectively, and 8 outputs. This is similar to the neural network architectures seen in the acrobot and cart-pole, as the neural network has a predetermined set of output actions, eight in this case, dependent on the index of the greatest output value.…”

Section: Mpc Quadrotormentioning

confidence: 99%

Verification of Closed-loop Systems with Neural Network Controllers

Lopez¹,

Musau²,

Tran³

et al.

EPiC Series in Computing

View full text Add to dashboard Cite

This benchmark suite presents a detailed description of a series of closed-loop control systems with artificial neural network controllers. In many applications, feed-forward neural networks are heavily involved in the implementation of controllers by learning and representing control laws through several methods such as model predictive control (MPC) and reinforcement learning (RL). The type of networks that we consider in this manuscript are feed-forward neural networks consisting of multiple hidden layers with ReLU activation functions and a linear activation function in the output layer. While neural network con- trollers have been able to achieve desirable performance in many contexts, they also present a unique challenge in that it is difficult to provide any guarantees about the correctness of their behavior or reason about the stability a system that employs their use. Thus, from a controls perspective, it is necessary to verify them in conjunction with their corresponding plants in closed-loop. While there have been a handful of works proposed towards the verification of closed-loop systems with feed-forward neural network controllers, this area still lacks attention and a unified set of benchmark examples on which verification techniques can be evaluated and compared. Thus, to this end, we present a range of closed-loop control systems ranging from two to six state variables, and a range of controllers with sizes in the range of eleven neurons to a few hundred neurons in more complex systems.

show abstract

Reasoning about safety of learning-enabled components in autonomous cyber-physical systems

Cited by 41 publications

References 14 publications

Survey on Scenario-Based Safety Assessment of Automated Vehicles

Survey on Scenario-Based Safety Assessment of Automated Vehicles

An Iterative Scheme of Safe Reinforcement Learning for Nonlinear Systems via Barrier Certificate Generation

Verification of Closed-loop Systems with Neural Network Controllers

Contact Info

Product

Resources

About