Towards Proving the Adversarial Robustness of Deep Neural Networks

Katz, Guy; Barrett, Clark; Dill, David L.; Julian, Kyle D.; Kochenderfer, Mykel J.

doi:10.4204/eptcs.257.3

Cited by 92 publications

(63 citation statements)

References 13 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, testing can not provide any safety guarantee in general. There are also attempts to formally verify certain safety properties against the DNN to provide certain safety guarantees [18], [20], [21], [47].…”

Section: Related Workmentioning

confidence: 99%

Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing

Wang

Dong

Sun

et al. 2019

2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

178

155

View full text Add to dashboard Cite

Deep neural networks (DNN) have been shown to be useful in a wide range of applications. However, they are also known to be vulnerable to adversarial samples. By transforming a normal sample with some carefully crafted human imperceptible perturbations, even highly accurate DNN make wrong decisions. Multiple defense mechanisms have been proposed which aim to hinder the generation of such adversarial samples. However, a recent work show that most of them are ineffective. In this work, we propose an alternative approach to detect adversarial samples at runtime. Our main observation is that adversarial samples are much more sensitive than normal samples if we impose random mutations on the DNN. We thus first propose a measure of 'sensitivity' and show empirically that normal samples and adversarial samples have distinguishable sensitivity. We then integrate statistical hypothesis testing and model mutation testing to check whether an input sample is likely to be normal or adversarial at runtime by measuring its sensitivity. We evaluated our approach on the MNIST and CIFAR10 datasets. The results show that our approach detects adversarial samples generated by state-of-the-art attacking methods efficiently and accurately.

show abstract

Section: Related Workmentioning

confidence: 99%

Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing

Wang

Dong

Sun

et al. 2019

2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

178

155

View full text Add to dashboard Cite

show abstract

“…Decision problems are further subdivided into verification and falsification, which seek a complete proof and counterexamples by best effort, respectively. Related works of verification are global safety [56], local safety [55], (ǫ, δ)-robustness [40], and (x, η, δ)safe [31]. Global safety is output bound, and local safety is the consistency of inference among close data points.…”

Section: Related Work and Research Directions For Verification Of Mamentioning

confidence: 99%

Engineering problems in machine learning systems

Kuwajima

Yasuoka²,

Nakae³

2020

Mach Learn

View full text Add to dashboard Cite

Fatal accidents are a major issue hindering the wide acceptance of safety-critical systems that employ machine learning and deep learning models, such as automated driving vehicles. In order to use machine learning in a safety-critical system, it is necessary to demonstrate the safety and security of the system through engineering processes. However, thus far, no such widely accepted engineering concepts or frameworks have been established for these systems. The key to using a machine learning model in a deductively engineered system is decomposing the data-driven training of machine learning models into requirement, design, and verification, particularly for machine learning models used in safety-critical systems. Simultaneously, open problems and relevant technical fields are not organized in a manner that enables researchers to select a theme and work on it. In this study, we identify, classify, and explore the open problems in engineering (safety-critical) machine learning systems -that is, in terms of requirement, design, and verification of machine learning models and systems -as well as discuss related works and research directions, using automated driving vehicles as an example. Our results show that machine learning models are characterized by a lack of requirements specification, lack of design specification, lack of interpretability, and lack of robustness. We also perform a gap analysis on a conventional system quality standard SQuARE with the characteristics of machine learning models to study quality models for machine learning systems. We find that a lack of requirements specification and lack of robustness have the greatest impact on conventional quality models.Preprint. Work in progress.

show abstract

“…Moreover, as described in Section I, an approach of encoding a DNN model in a formula and verifying it by determining the satisfiability of the formula has been proposed in recent years [15] [16] [17] [18] [14] [19]. It can be said that the method for verifying the DTEM proposed in this paper also takes this approach.…”

Section: Related Workmentioning

confidence: 99%

Formal Verification of a Decision-Tree Ensemble Model and Detection of Its Violation Ranges

Sato

Kuruma

Nakagawa

et al. 2020

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

As one type of machine-learning model, a "decisiontree ensemble model" (DTEM) is represented by a set of decision trees. A DTEM is mainly known to be valid for structured data; however, like other machine-learning models, it is difficult to train so that it returns the correct output value for any input value. Accordingly, when a DTEM is used in regard to a system that requires reliability, it is important to comprehensively detect input values that lead to malfunctions of a system (failures) during development and take appropriate measures. One conceivable solution is to install an input filter that controls the input to the DTEM, and to use separate software to process input values that may lead to failures. To develop the input filter, it is necessary to specify the filtering condition of the input value that leads to the malfunction of the system. Given that necessity, in this paper, we propose a method for formally verifying a DTEM and, according to the result of the verification, if an input value leading to a failure is found, extracting the range in which such an input value exists. The proposed method can comprehensively extract the range in which the input value leading to the failure exists; therefore, by creating an input filter based on that range, it is possible to prevent the failure occurring in the system. In this paper, the algorithm of the proposed method is described, and the results of a case study using a dataset of house prices are presented. On the basis of those results, the feasibility of the proposed method is demonstrated, and its scalability is evaluated.

show abstract

Towards Proving the Adversarial Robustness of Deep Neural Networks

Cited by 92 publications

References 13 publications

Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing

Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing

Engineering problems in machine learning systems

Formal Verification of a Decision-Tree Ensemble Model and Detection of Its Violation Ranges

Contact Info

Product

Resources

About