A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems

Abstract

Autonomous cyber-physical systems (CPS) can improve safety and efficiency for safety-critical applications, but require rigorous testing before deployment. The complexity of these systems often precludes the use of formal verification and real-world testing can be too dangerous during development. Therefore, simulation-based techniques have been developed that treat the system under test as a black box operating in a simulated environment. Safety validation tasks include finding disturbances in the environment that cause the system to fail (falsification), finding the most-likely failure, and estimating the probability that the system fails. Motivated by the prevalence of safety-critical artificial intelligence, this work provides a survey of state-of-the-art safety validation techniques for CPS with a focus on applied algorithms and their modifications for the safety validation problem. We present and discuss algorithms in the domains of optimization, path planning, reinforcement learning, and importance sampling. Problem decomposition techniques are presented to help scale algorithms to large state spaces, which are common for CPS. A brief overview of safety-critical applications is given, including autonomous vehicles and aircraft collision avoidance systems. Finally, we present a survey of existing academic and commercially available safety validation tools.

Verifying Inverse Model Neural Networks

“…Sampling can be used to assess the quality of a neural network; neural networks are typically evaluated on a test set after training (Goodfellow et al, 2016). Statistical approaches to verification (Corso et al, 2021) have been used for a wide variety of applications such as autonomous driving (Huang et al, 2018) and aviation (Zakrzewski, 2004). Zakrzewski (2004) investigates the number of samples required to estimate the failure probability with high confidence, considering Bayesian and non-Bayesian settings.…”

“…If sampling is performed using a distribution that is slightly wrong, the claims of safety assurance fall apart. Consequently, other methods obtain guarantees using sampling coverage metrics over the input space, which again scale exponentially with the dimension of the input domain (Corso et al, 2021). Given the difficulties of sampling-based safety guarantees, we believe formal guarantees which hold over the entire input domain are more compelling.…”

A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation

“…In contrast, we focus on a more specific topic, scenario-based ADS testing in high-fidelity simulation in much more profound depth. [141] reviews widely used algorithms for black-box safety validation. It focuses on presenting the general algorithms, while we present the algorithms along with other components (e.g., simulators, systems, scenario parameters, and testing objectives) in the context of ADS testing.…”

Testing Robot System Safety by creating Hazardous Human Worker Behavior in Simulation

“…But as simulation models are often complex and safety-critical states rare, creating interesting and critical test scenarios becomes a challenge [24]. Often, one uses algorithms for search, optimization, or machine-learning to find simulation conditions that expose hazards of failures [25]. Different heuristics (e.g.…”