An Abstraction-Refinement Approach to Verification of Artificial Neural Networks

Pulina, Luca; Tacchella, Armando

doi:10.1007/978-3-642-14295-6_24

Cited by 257 publications

(216 citation statements)

References 13 publications

Supporting

Mentioning

200

Contrasting

Order By: Relevance

“…And of course all these issues are laid on top of the standard problem of proving that a given software artifact does in fact correctly implement, say, a reinforcement learning algorithm of the intended type. Some work has been done on verifying neural network applications (Pulina and Tacchella 2010;Taylor 2006;Schumann and Liu 2010) and the notion of partial programs (Andre and Russell 2002;Spears 2006) allows the designer to impose arbitrary structural constraints on behavior, but much remains to be done before it will be possible to have high confidence that a learning agent will learn to satisfy its design criteria in realistic contexts. Validity A verification theorem for an agent design has the form, "If environment satisfies assumptions ϕ then behavior satisfies requirements ψ."…”

Section: Professional Ethicsmentioning

confidence: 99%

Research Priorities for Robust and Beneficial Artificial Intelligence

2015

View full text Add to dashboard Cite

Articles WINTER 2015 105A rtificial intelligence (AI) research has explored a variety of problems and approaches since its inception, but for the last 20 years or so has been focused on the problems surrounding the construction of intelligent agentssystems that perceive and act in some environment. In this context, the criterion for intelligence is related to statistical and economic notions of rationality -colloquially, the ability to make good decisions, plans, or inferences. The adoption of probabilistic representations and statistical learning methods has led to a large degree of integration and crossfertilization between AI, machine learning, statistics, control theory, neuroscience, and other fields. The establishment of shared theoretical frameworks, combined with the availability of data and processing power, has yielded remarkable successes in various component tasks such as speech recognition, image classification, autonomous vehicles, machine translation, legged locomotion, and question-answering systems. Copyright AI MAGAZINEAs capabilities in these areas and others cross the threshold from laboratory research to economically valuable technologies, a virtuous cycle takes hold whereby even small improvements in performance have significant economic value, prompting greater investments in research. There is now a broad consensus that AI research is progressing steadily, and that its impact on society is likely to increase. The potential benefits are huge, since everything that civilization has to offer is a product of human intelligence; we cannot predict what we might achieve when this intelligence is magnified by the tools AI may provide, but the eradication of disease and poverty is not unfathomable. Because of the great potential of AI, it is valuable to investigate how to reap its benefits while avoiding potential pitfalls.Progress in AI research makes it timely to focus research not only on making AI more capable, but also on maximizing the societal benefit of AI. Such considerations motivated the AAAI 2008-09 Presidential Panel on Long-Term AI Futures (Horvitz and Selman 2009) and other projects and community efforts on AI's future impacts. These constitute a significant expansion of the field of AI itself, which up to now has focused largely on techniques that are neutral with respect to purpose. The present document can be viewed as a natural continuation of these efforts, focusing on identifying research directions that can help maximize the societal benefit of AI. This research is by necessity interdisciplinary, because it involves both society and AI. It ranges from economics, law, and philosophy to computer security, formal methods, and, of course, various branches of AI itself. The focus is on delivering AI that is beneficial to society and robust in the sense that the benefits are guaranteed: our AI systems must do what we want them to do.This article was drafted with input from the attendees of the 2015 conference The Future of AI: Opportunities and Challenges (see Acknowledgements),...

show abstract

Section: Professional Ethicsmentioning

confidence: 99%

Research Priorities for Robust and Beneficial Artificial Intelligence

2015

View full text Add to dashboard Cite

show abstract

“…In [13] we showed that an abstraction mechanism can be devised to yield consistent overapproximations of concrete networks, i.e., once the abstract MLP is proven to be safe, the same holds true for the concrete one. We now outline the abstraction mechanism whose details can be found in [13].…”

Section: Abstractionmentioning

confidence: 76%

“…Abstraction is a key enabler for verification because MLPs are compositions of non-linear and transcendental real-valued functions, and the theories to handle such functions are undecidable [17]. In [13] we showed that an abstraction mechanism can be devised to yield consistent overapproximations of concrete networks, i.e., once the abstract MLP is proven to be safe, the same holds true for the concrete one. We now outline the abstraction mechanism whose details can be found in [13].…”

Section: Abstractionmentioning

confidence: 86%

“…In [13] we proposed an abstraction of MLPs to corresponding Boolean combinations of linear constraints, following the abstract interpretation framework described in [27]. Abstraction is a key enabler for verification because MLPs are compositions of non-linear and transcendental real-valued functions, and the theories to handle such functions are undecidable [17].…”

Section: Abstractionmentioning

confidence: 99%

“…We chose this kind of network because, albeit of its relatively simple topology, MLPs can in principle approximate every non-linear real-valued function of n real-valued inputs, as stated by the Universal Approximation Theorem for ANNs [12]. The core of NeVer is composed by an abstraction-refinement mechanism to verify MLPs proposed, for the first time, in [13]. The intuition behind this mechanism is that the abstract MLP provides a sound overapproximation of the concrete one, i.e.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

NeVer: a tool for artificial neural networks verification

Pulina

Tacchella

2011

Ann Math Artif Intell

Self Cite

View full text Add to dashboard Cite

The adoption of Artificial Neural Networks (ANNs) in safety-related ap- plications is often avoided because it is difficult to rule out possible misbehaviors with traditional analytical or probabilistic techniques. In this paper we present NeVer, our tool for checking safety of ANNs. NeVer encodes the problem of verifying safety of ANNs into the problem of satisfying corresponding Boolean combinations of linear arithmetic constraints. We describe the main verification algorithm and the structure of NeVer. We present also empirical results confirming the effectiveness of NeVer on realistic case studies

show abstract

DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks

Gopinath

Katz

Păsăreanu

et al. 2018

Lecture Notes in Computer Science

126

View full text Add to dashboard Cite

Deep neural networks have become widely used, obtaining remarkable results in domains such as computer vision, speech recognition, natural language processing, audio recognition, social network filtering, machine translation, and bio-informatics, where they have produced results comparable to human experts. However, these networks can be easily "fooled" by adversarial perturbations: minimal changes to correctly-classified inputs, that cause the network to misclassify them. This phenomenon represents a concern for both safety and security, but it is currently unclear how to measure a network's robustness against such perturbations. Existing techniques are limited to checking robustness around a few individual input points, providing only very limited guarantees. We propose a novel approach for automatically identifying safe regions of the input space, within which the network is robust against adversarial perturbations. The approach is data-guided, relying on clustering to identify well-defined geometric regions as candidate safe regions. We then utilize verification techniques to confirm that these regions are safe or to provide counter-examples showing that they are not safe. We also introduce the notion of targeted robustness which, for a given target label and region, ensures that a NN does not map any input in the region to the target label. We evaluated our technique on the MNIST dataset and on a neural network implementation of a controller for the next-generation Airborne Collision Avoidance System for unmanned aircraft (ACAS Xu). For these networks, our approach identified multiple regions which were completely safe as well as some which were only safe for specific labels. It also discovered several adversarial perturbations of interest.

show abstract

An Abstraction-Refinement Approach to Verification of Artificial Neural Networks

Cited by 257 publications

References 13 publications

Research Priorities for Robust and Beneficial Artificial Intelligence

Research Priorities for Robust and Beneficial Artificial Intelligence

NeVer: a tool for artificial neural networks verification

DeepSafe: A Data-Driven Approach for Assessing Robustness of Neural Networks

Contact Info

Product

Resources

About