Model-Reuse Attacks on Deep Learning Systems

Abstract

Many of today's machine learning (ML) systems are built by reusing an array of, often pre-trained, primitive models, each fulfilling distinct functionality (e.g., feature extraction). The increasing use of primitive models significantly simplifies and expedites the development cycles of ML systems. Yet, because most of such models are contributed and maintained by untrusted sources, their lack of standardization or regulation entails profound security implications, about which little is known thus far.In this paper, we demonstrate that malicious primitive models pose immense threats to the security of ML systems. We present a broad class of model-reuse attacks wherein maliciously crafted models trigger host ML systems to misbehave on targeted inputs in a highly predictable manner. By empirically studying four deep learning systems (including both individual and ensemble systems) used in skin cancer screening, speech recognition, face verification, and autonomous steering, we show that such attacks are (i) effective -the host systems misbehave on the targeted inputs as desired by the adversary with high probability, (ii) evasive -the malicious models function indistinguishably from their benign counterparts on non-targeted inputs, (iii) elastic -the malicious models remain effective regardless of various system design choices and tuning strategies, and (iv) easy -the adversary needs little prior knowledge about the data used for system tuning or inference. We provide analytical justification for the effectiveness of model-reuse attacks, which points to the unprecedented complexity of today's primitive models. This issue thus seems fundamental to many ML systems. We further discuss potential countermeasures and their challenges, which lead to several promising research directions.

A Review of Deep Learning Security and Privacy Defensive Techniques

“…Ji et al [44] proposed classification of security threats for Deep Learning in 3 different angles, which influence classifieds, security breaches, and privacy of attacks.…”

“…e software bugs of the systems that hosted Deep Learning applications on its operating system can be hijacked due to remote compromise and application bugs [44,82]. is mostly happens when the system is connected with the cloud system and the Deep Learning applications are also running on that cloud-based system.…”

Towards Adversarial Malware Detection

“…Conversely, if the attack is targeted to only have few samples misclassified at test time, then it is named as a poisoning integrity attack [7,8,11,15,17,45,46,49,50,69]. We also include backdoor attacks into this category, as they also aim to cause specific misclassifications at test time by compromising the training process of the learning algorithm [22,43,47,57]. Their underlying idea is to compromise a model during the training phase or, more generally, at design time (this may include, e.g., also modifications to the architecture of a deep network by addition of specific layers or neurons), with the goal of enforcing the model to classify backdoored samples at test time as desired by the attacker.…”

Making Targeted Black-box Evasion Attacks Effective and Efficient

“…They show that their attack performance degrades if several layers of the student models are fine-tuned. Ji et al [23] maliciously train pre-trained models in order to implement model-reuse attacks against ML systems without knowing the developer's dataset or fine-tuning strategies. Hashemi et al [17] query the target model with images that come from a similar distribution as the training images of the target model, augment the dataset with random noise and use this augmented dataset to train a substitute model.…”