Detecting AI Trojans Using Meta Neural Analysis

Abstract

Machine learning models, especially neural networks (NNs), have achieved outstanding performance on diverse and complex applications. However, recent work has found that they are vulnerable to Trojan attacks where an adversary trains a corrupted model with poisoned data or directly manipulates its parameters in a stealthy way. Such Trojaned models can obtain good performance on normal data during test time while predict incorrectly on the adversarially manipulated data samples. This paper aims to develop ways to detect Trojaned models. We mainly explore the idea of meta neural analysis, a technique involving training a meta NN model that can be used to predict whether or not a target NN model has certain properties. We develop a novel pipeline Meta Neural Trojaned model Detection (MNTD) system to predict if a given NN is Trojaned via meta neural analysis on a set of trained shadow models.We propose two ways to train the meta-classifier without knowing the Trojan attacker's strategies. The first one, oneclass learning, will fit a novel detection meta-classifier using only benign neural networks. The second one, called jumbo learning, will approximate a general distribution of Trojaned models and sample a "jumbo" set of Trojaned models to train the metaclassifier and evaluate on the unseen Trojan strategies. Extensive experiments demonstrate the effectiveness of MNTD in detecting different Trojan attacks in diverse areas such as vision, speech, tabular data, and natural language processing. We show that MNTD reaches an average of 97% detection AUC (Area Under the ROC Curve) score and outperforms existing approaches. Furthermore, we design and evaluate MNTD system to defend against strong adaptive attackers who have exactly the knowledge of the detection, which demonstrates the robustness of MNTD.

Backdoor Attacks on Image Classification Models in Deep Neural Networks

“…It is conducted at inference time, like an adversarial attack. Recent research shows that shallow convolutional layers learn common properties among images [81] , such as edge and colour. Inspired by this, attackers propose to utilize a colour stripe pattern generated by modulating LED in a specialized waveform as a trigger.…”

Baseline Pruning-Based Approach to Trojan Detection in Neural Networks

“…The research interest increased as IARPA and Defense Advanced Research Projects Agency (DARPA) announced the TrojAI [2] and Guaranteeing AI Robustness Against Deception (GARD) [4] programs in 2019. With more research efforts invested into designs of trojan detectors [5], [6], [7], there is a need to establish a baseline method that is simple, but generally applicable, and provides results that are better than a chance [8].…”

Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses

“…Different from the detection pipelines discussed above, MNTD (Xu et al 2021) predicts whether a model is backdoored by examining its behavior on carefully crafted inputs. MNTD first generates a battery of benign and backdoored models.…”

“…The query set is then jointly optimized with the parameters of the meta-classifier to obtain a high accuracy meta-classifier. Interestingly, this approach appears to detect attacks on architectures outside of the ensemble used to train the meta-classifier (Xu et al 2021). Huang et al ( 2020) define a "one-pixel" signature of a network, which is the collection of single-pixel adversarial perturbations that most effectively impact the label of a collection of images.…”