Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization

Muñoz-González, Luis; Biggio, Battista; Demontis, Ambra; Paudice, Andrea; Wongrassamee, Vasin; Lupu, Emil; Roli, Fabio

doi:10.1145/3128572.3140451

Cited by 435 publications

(397 citation statements)

References 24 publications

Supporting

Mentioning

395

Contrasting

Order By: Relevance

“…Two primary threat models are proposed in literature. (i) Poisoning attacks, in which the adversary pollutes the training data to eventually compromise the ML systems [9,41,61,62]. Such attacks can be further categorized as targeted and untargeted attacks.…”

Section: Related Workmentioning

confidence: 99%

Model-Reuse Attacks on Deep Learning Systems

Zhang

et al. 2018

Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security

148

128

View full text Add to dashboard Cite

Many of today's machine learning (ML) systems are built by reusing an array of, often pre-trained, primitive models, each fulfilling distinct functionality (e.g., feature extraction). The increasing use of primitive models significantly simplifies and expedites the development cycles of ML systems. Yet, because most of such models are contributed and maintained by untrusted sources, their lack of standardization or regulation entails profound security implications, about which little is known thus far.In this paper, we demonstrate that malicious primitive models pose immense threats to the security of ML systems. We present a broad class of model-reuse attacks wherein maliciously crafted models trigger host ML systems to misbehave on targeted inputs in a highly predictable manner. By empirically studying four deep learning systems (including both individual and ensemble systems) used in skin cancer screening, speech recognition, face verification, and autonomous steering, we show that such attacks are (i) effective -the host systems misbehave on the targeted inputs as desired by the adversary with high probability, (ii) evasive -the malicious models function indistinguishably from their benign counterparts on non-targeted inputs, (iii) elastic -the malicious models remain effective regardless of various system design choices and tuning strategies, and (iv) easy -the adversary needs little prior knowledge about the data used for system tuning or inference. We provide analytical justification for the effectiveness of model-reuse attacks, which points to the unprecedented complexity of today's primitive models. This issue thus seems fundamental to many ML systems. We further discuss potential countermeasures and their challenges, which lead to several promising research directions.

show abstract

Section: Related Workmentioning

confidence: 99%

Model-Reuse Attacks on Deep Learning Systems

Zhang

et al. 2018

Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security

148

128

View full text Add to dashboard Cite

show abstract

“…Other than adversarial examples, we could also leverage data poisoning attacks [65][66][67][68][69][70][71][72] to defend against inference attacks. Specifically, an attacker needs to train an ML classifier in inference attacks.…”

Section: Data Poisoning Attacks Based Defensesmentioning

confidence: 99%

Defending Against Machine Learning Based Inference Attacks via Adversarial Examples: Opportunities and Challenges

Jia

Gong

2020

Adaptive Autonomous Secure Cyber Systems

View full text Add to dashboard Cite

As machine learning (ML) becomes more and more powerful and easily accessible, attackers increasingly leverage ML to perform automated large-scale inference attacks in various domains. In such an ML-equipped inference attack, an attacker has access to some data (called public data) of an individual, a software, or a system; and the attacker uses an ML classifier to automatically infer their private data. Inference attacks pose severe privacy and security threats to individuals and systems. Inference attacks are successful because private data are statistically correlated with public data, and ML classifiers can capture such statistical correlations. In this chapter, we discuss the opportunities and challenges of defending against ML-equipped inference attacks via adversarial examples. Our key observation is that attackers rely on ML classifiers in inference attacks. The adversarial machine learning community has demonstrated that ML classifiers have various vulnerabilities. Therefore, we can turn the vulnerabilities of ML into defenses against inference attacks. For example, ML classifiers are vulnerable to adversarial examples, which add carefully crafted noise to normal examples such that an ML classifier makes predictions for the examples as we desire. To defend against inference attacks, we can add carefully crafted noise into the public data to turn them into adversarial examples, such that attackers' classifiers make incorrect predictions for the private data. However, existing methods to construct adversarial examples are insufficient because they did not consider the unique challenges and requirements for the crafted noise at defending against inference attacks. In this chapter, we take defending against inference attacks in online social networks as an example to illustrate the opportunities and challenges.

show abstract

“…An important property of the regions determined by Algorithm 2 is stated by the following proposition. where L k (R) defined in (6). In other words, the LiDAR ray with angle θ k intersects the same obstacle edge regardless of the robot position.…”

Section: Partitioning the Workpacementioning

confidence: 99%

“…Motivated by the urgency to study the safety, reliability, and potential problems that can rise and impact the society by the deployment of AI-enabled systems in the real world, several works in the literature focused on the problem of designing deep neural networks that are robust to the so-called adversarial examples [2][3][4][5][6][7][8]. Unfortunately, these techniques focus mainly on the robustness of the learning algorithm with respect to data outliers without providing guarantees in terms of safety and reliability of the decisions taken by these neural networks.…”

Section: Introductionmentioning

confidence: 99%

Formal verification of neural network controlled autonomous systems

Sun

Khedr

Shoukry

2019

Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control

105

View full text Add to dashboard Cite

In this paper, we consider the problem of formally verifying the safety of an autonomous robot equipped with a Neural Network (NN) controller that processes LiDAR images to produce control actions. Given a workspace that is characterized by a set of polytopic obstacles, our objective is to compute the set of safe initial conditions such that a robot trajectory starting from these initial conditions is guaranteed to avoid the obstacles. Our approach is to construct a finite state abstraction of the system and use standard reachability analysis over the finite state abstraction to compute the set of the safe initial states. The first technical problem in computing the finite state abstraction is to mathematically model the imaging function that maps the robot position to the LiDAR image. To that end, we introduce the notion of imaging-adapted sets as partitions of the workspace in which the imaging function is guaranteed to be affine. Based on this notion, and resting on efficient algorithms in the literature of computational geometry, we develop a polynomialtime algorithm to partition the workspace into imaging-adapted sets along with computing the corresponding affine imaging functions. Given this workspace partitioning, a discrete-time linear dynamics of the robot, and a pre-trained NN controller with Rectified Linear Unit (ReLU) nonlinearity, the second technical challenge is to analyze the behavior of the neural network. To that end, and thanks to the ReLU functions being piecewise linear functions, we utilize a Satisfiability Modulo Convex (SMC) encoding to enumerate all the possible segments of different ReLUs. SMC solvers then use a Boolean satisfiability solver and a convex programming solver and decompose the problem into smaller subproblems. At each iteration, the Boolean satisfiability solver searches for a candidate assignment for the different ReLU segments while completely abstracting the robot dynamics. Convex programming is then used to check the feasibility of the proposed ReLU phases against the dynamic and imagining constraints, or generate succinct explanations for their infeasibility to reduce the search space. To accelerate this process, we develop a pre-processing algorithm that could rapidly prune the space feasible ReLU segments. Finally, we demonstrate the efficiency of the proposed algorithms using numerical simulations with increasing complexity of the neural network controller.

show abstract

Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization

Cited by 435 publications

References 24 publications

Model-Reuse Attacks on Deep Learning Systems

Model-Reuse Attacks on Deep Learning Systems

Defending Against Machine Learning Based Inference Attacks via Adversarial Examples: Opportunities and Challenges

Formal verification of neural network controlled autonomous systems

Contact Info

Product

Resources

About