Odyssey: Creation, Analysis and Detection of Trojan Models

Abstract

Along with the success of deep neural network (DNN) models in solving various real world problems, rise the threats to these models that aim to degrade their integrity. Trojan attack is one of the recent variant of data poisoning attacks that involves manipulation or modification of the model to act balefully. This can occur when an attacker interferes with the training pipeline by inserting triggers into some of the training samples and trains the model to act maliciously only for samples that are stamped with trigger. Since the knowledge of such triggers is only privy to the attacker, detection of Trojan behaviour is a challenge task. Unlike any of the existing Trojan detectors, a robust detector should not rely on any assumption about Trojan attack. In this paper, we develop a detector based upon the analysis of intrinsic properties of DNN that could get affected by a Trojan attack. To have a comprehensive study, we propose , Odysseus the largest Trojan dataset with over 3,000 trained DNN models, both clean and Trojan. It covers a large spectrum of attacks; generated by leveraging the versatility in designing a trigger and mapping (source to target class) type. Our findings reveal that Trojan attacks affect the classifier margin and shape of decision boundary around the manifold of the clean data. Combining these two factors leads to an efficient Trojan detector; operates irrespective of any knowledge of the Trojan attack; that sets the first baseline for this task with accuracy above 83%. The Odysseus dataset along with the Trojan detector can be downloaded here

The Prominence of Artificial Intelligence in COVID-19

“…The top algorithms of it are: Random Forest, Decision Trees, Logistic Regression, Support Vector Machines. Convolutional Neural Networks (CNN) by Al-Saffar et al (2017); Edraki et al (2020), Recurrent Neural Network (RNN) by Sak et al (2014), Long Short-Term Memory (LSTM) by Sak et al (2014), Multilayer perceptron (MLP). These are the types of DL that are generally trained as supervised methods.…”

MISA: Online Defense of Trojaned Models using Misattributions

“…The attack is insidious since the Trojan trigger is only known to the attacker; the model outputs the correct label when the trigger is absent. Other state-of-the-art Trojan insertion methods are proposed in [9,22,34,51,4]. Inserting Trojans using transfer learning [46] or retraining [25] has been demonstrated.…”

Detecting Trojaned DNNs Using Counterfactual Attributions

“…The state-of-the-art Trojan insertion methods [12,32,44,63,17,8,17] use a minuscule amount of data poisoned with the Trojan trigger pattern (e.g., a local patch, a filter with specific settings). Alternative methods inject Trojans through transfer learning [58], retraining a DNN [35], direct manipulation of DNN weights [11,43], or addition of malicious modules [50].…”