Validation of Image-Based Neural Network Controllers through Adaptive Stress Testing

Abstract

Neural networks have become state-of-the-art for computer vision problems because of their ability to efficiently model complex functions from large amounts of data. While neural networks can be shown to perform well empirically for a variety of tasks, their performance is difficult to guarantee. Neural network verification tools have been developed that can certify robustness with respect to a given input image; however, for neural network systems used in closed-loop controllers, robustness with respect to individual images does not address multi-step properties of the neural network controller and its environment. Furthermore, neural network systems interacting in the physical world and using natural images are operating in a black-box environment, making formal verification intractable. This work combines the adaptive stress testing (AST) framework with neural network verification tools to search for the most likely sequence of image disturbances that cause the neural network controlled system to reach a failure. An autonomous aircraft taxi application is presented, and results show that the AST method finds failures with more likely image disturbances than baseline methods. Further analysis of AST results revealed an explainable cause of the failure, giving insight into the problematic scenarios that should be addressed.

A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems

“…Monte Carlo tree search (MCTS) is an online planning algorithm for sequential decision making that has seen success for long time-horizon problems (Silver et al, 2016), making it useful for safety validation (Delmas et al, 2019;Julian et al, 2020;Lee et al, 2020;Moss et al, 2020;Wicker et al, 2018;Zhang et al, 2018). Monte Carlo tree search uses online planning to determine the best actions to take from a starting state to maximize reward.…”

“…Validation algorithms search for violations of a minimum altitude from various initial conditions. Julian et al (2020) perform safety validation on a neural network controller with camera inputs for aircraft taxiing.…”

ZoPE: A Fast Optimizer for ReLU Networks with Low-Dimensional Inputs

“…For example, ERAN's parameters were based off of the VNN20 competition as found at https://github.com/GgnDpSngh/ERAN-VNN-COMP/blob/master/tf_verify/run_ acasxu.sh. The source code for for MARABOU, 5 NNENUM, 6 ERAN, 7 and our optimizer ZOPE 8 is available for free online.…”

Sense–Assess–eXplain (SAX): Building Trust in Autonomous Vehicles in Challenging Real-World Driving Scenarios

“…In the broader community and as exhibited in focused sessions [7], [8] we find that the works of [9]- [22] have a bearing on this research agenda. In particular, works dealing with validation and proving safety [15], [20], critical scenarios and hazardous events [10], [21], human values and comfort [11], [19], human-understandable descriptions [22], safety-oriented architectures [17], [18], and mitigating hazardous events [14] all related to our objectives. Furthermore, from a robotics perspective, the role of perceptual components in safety systems [9] as well as demonstrators [12], simulators [13] and datasets [16] for tackling these research challenges are all approaches that inspire the work presented.…”