VerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems

Dreossi, Tommaso; Fremont, Daniel J.; Ghosh, Sandip; Kim, Edward; Ravanbakhsh, Hadi; Vazquez-Chanlatte, Marcell; Seshia, Sanjit A.

doi:10.1007/978-3-030-25540-4_25

Cited by 161 publications

(136 citation statements)

References 21 publications

(40 reference statements)

Supporting

Mentioning

127

Contrasting

Order By: Relevance

“…7. The results confirmed the importance of model and color (compare (2) to (7)), as well as angle (compare (5) to (6)), but also suggested that being close to the camera could be the relevant aspect of the car's local position. We confirmed this with a final round of scenarios (compare (5) and (8)), which also showed that the effect of car model is small among scenes where the car is close to the camera (compare (4) and (9)).…”

Section: Training On Rare Eventssupporting

confidence: 75%

“…In future work we plan to conduct experiments applying Scenic to a variety of additional domains, applications, and simulators. For example, we have integrated Scenic as the environment modeling language for VerifAI, a tool for the design and analysis of AI-based systems [7], and used it to generate seed inputs for temporal-logic falsification of an automated collision-avoidance system. We have also interfaced Scenic to the X-Plane flight simulator [36] in order to test ML-based aircraft navigation systems, and to the CARLA driving simulator [6] for scenarios requiring more control than GTAV provides.…”

Section: Resultsmentioning

confidence: 99%

“…See Appendix D for details. 7 Recall is much higher on T overlap , meaning the false-negative rate is better; this is presumably because all the T overlap images have exactly 2 cars and are in that sense easier than the T matrix images, which can have many cars. We selected one scene from our first experiment, consisting of a single car viewed from behind at a slight angle, which M generic wrongly classified as three cars (thus having 33.3% precision and 100% recall).…”

Section: Training On Rare Eventsmentioning

confidence: 99%

“…To demonstrate this, we first wrote very general scenarios describing scenes of 1-4 cars (not counting the camera), specifying only that the cars face within 10 • of the road direction. We generated 1,000 images from each scenario, yielding a training set X generic of 4,000 images, and used 1 wiggle = (-10 deg, 10 deg) 2 ego = Car with roadDeviation wiggle 3 c = Car visible, 4 with roadDeviation resample(wiggle) 5 leftRight = Uniform(1.0, -1.0) * (1.25, 2.75) 6 Car beyond c by leftRight @ (4, 10), 7 with roadDeviation resample(wiggle) Figure 8. A scenario where one car partially occludes another.…”

Section: Testing Under Different Conditionsmentioning

confidence: 99%

See 3 more Smart Citations

Scenic: a language for scenario specification and scene generation

Fremont

Dreossi

Ghosh

et al. 2019

Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation

Self Cite

209

188

View full text Add to dashboard Cite

Figure 1. Three scenes generated from a single ∼20-line Scenic scenario representing bumper-to-bumper traffic. AbstractWe propose a new probabilistic programming language for the design and analysis of perception systems, especially those based on machine learning. Specifically, we consider the problems of training a perception system to handle rare events, testing its performance under different conditions, and debugging failures. We show how a probabilistic programming language can help address these problems by specifying distributions encoding interesting types of inputs and sampling these to generate specialized training and test sets. More generally, such languages can be used for cyberphysical systems and robotics to write environment models, an essential prerequisite to any formal analysis. In this pa- * These authors contributed equally to the paper. per, we focus on systems like autonomous cars and robots, whose environment is a scene, a configuration of physical objects and agents. We design a domain-specific language, Scenic, for describing scenarios that are distributions over scenes. As a probabilistic programming language, Scenic allows assigning distributions to features of the scene, as well as declaratively imposing hard and soft constraints over the scene. We develop specialized techniques for sampling from the resulting distribution, taking advantage of the structure provided by Scenic's domain-specific syntax. Finally, we apply Scenic in a case study on a convolutional neural network designed to detect cars in road images, improving its performance beyond that achieved by state-of-the-art synthetic data generation methods.

show abstract

Section: Training On Rare Eventssupporting

confidence: 75%

Section: Resultsmentioning

confidence: 99%

Section: Training On Rare Eventsmentioning

confidence: 99%

Section: Testing Under Different Conditionsmentioning

confidence: 99%

See 2 more Smart Citations

Scenic: a language for scenario specification and scene generation

Fremont

Dreossi

Ghosh

et al. 2019

Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation

Self Cite

209

188

View full text Add to dashboard Cite

show abstract

“…While formal verification is desirable, in practice, it can be difficult due to high system complexity, unavailability of source code, and limited capacity of existing verification tools. In addition, many systems have started incorporating machine learning components, which remain challenging to test or verify [13], [21].…”

Section: Introductionmentioning

confidence: 99%

Shield Synthesis for Real: Enforcing Safety in Cyber-Physical Systems

Wang

Deshmukh

et al. 2019

2019 Formal Methods in Computer Aided Design (FMCAD)

View full text Add to dashboard Cite

Cyber-physical systems are often safety-critical in that violations of safety properties may lead to catastrophes. We propose a method to enforce the safety of systems with real-valued signals by synthesizing a runtime enforcer called the shield. Whenever the system violates a property, the shield, composed with the system, makes correction instantaneously to ensure that no erroneous output is generated by the combined system. While techniques for synthesizing Boolean shields are well understood, they do not handle real-valued signals ubiquitous in cyber-physical systems, meaning their corrections may be either unrealizable or inefficient to compute in the real domain. We solve the realizability and efficiency problems by analyzing the compatibility of predicates defined over real-valued signals, and using the analysis result to constrain a two-player safety game used to synthesize the shield. We demonstrate the effectiveness of this method on a variety of applications, including an automotive powertrain control system.

show abstract

NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems

Tran

Yang

Lopez

et al. 2020

Lecture Notes in Computer Science

175

141

View full text Add to dashboard Cite

This paper presents the Neural Network Verification (NNV) software tool, a set-based verification framework for deep neural networks (DNNs) and learning-enabled cyber-physical systems (CPS). The crux of NNV is a collection of reachability algorithms that make use of a variety of set representations, such as polyhedra, star sets, zonotopes, and abstract-domain representations. NNV supports both exact (sound and complete) and over-approximate (sound) reachability algorithms for verifying safety and robustness properties of feed-forward neural networks (FFNNs) with various activation functions. For learning-enabled CPS, such as closed-loop control systems incorporating neural networks, NNV provides exact and over-approximate reachability analysis schemes for linear plant models and FFNN controllers with piecewise-linear activation functions, such as ReLUs. For similar neural network control systems (NNCS) that instead have nonlinear plant models, NNV supports overapproximate analysis by combining the star set analysis used for FFNN controllers with zonotope-based analysis for nonlinear plant dynamics building on CORA. We evaluate NNV using two real-world case studies: the first is safety verification of ACAS Xu networks, and the second deals with the safety verification of a deep learning-based adaptive cruise control system.

show abstract

VerifAI: A Toolkit for the Formal Design and Analysis of Artificial Intelligence-Based Systems

Cited by 161 publications

References 21 publications

Scenic: a language for scenario specification and scene generation

Scenic: a language for scenario specification and scene generation

Shield Synthesis for Real: Enforcing Safety in Cyber-Physical Systems

NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems

Contact Info

Product

Resources

About