Detecting Adversarial Perturbations Through Spatial Behavior in Activation Spaces

Abstract

Neural network based classifiers are still prone to manipulation through adversarial perturbations. State of the art attacks can overcome most of the defense or detection mechanisms suggested so far, and adversaries have the upper hand in this arms race.Adversarial examples are designed to resemble the normal input from which they were constructed, while triggering an incorrect classification. This basic design goal leads to a characteristic spatial behavior within the context of Activation Spaces, a term coined by the authors to refer to the hyperspaces formed by the activation values of the network's layers. Within the output of the first layers of the network, an adversarial example is likely to resemble normal instances of the source class, while in the final layers such examples will diverge towards the adversary's target class. The steps below enable us to leverage this inherent "shift" from one class to another in order to form a novel adversarial example detector. We construct Euclidian spaces out of the activation values of each of the deep neural network layers. Then, we induce a set of knearest neighbor classifiers (k-NN), one per activation space of each neural network layer, using the non-adversarial examples. We leverage those classifiers to produce a sequence of class labels for each nonperturbed input sample and estimate the a priori probability for a class label change between one activation space and another. Training our detector with only normal input follows the principles of anomaly detection, aiming to make our method future ready for as yet unknown attack methods. During the detection phase we compute a sequence of classification labels for each input using the trained classifiers. We then estimate the likelihood of those classification sequences and show that adversarial sequences are far less likely than normal ones.We evaluated our detection method against the state of the art C&W attack method, using two image classification datasets (MNIST, CIFAR-10) which are commonly used for adversarial evaluation. Our evaluation results show that our detector achieves an AUC of 0.95 for the CIFAR-10 dataset, with only a marginal increase in the computational complexity.

Adversarial Attacks Against Deep Generative Models on Data: A Survey

“…The aim of activation output clustering is to detect anomalous input by analyzing the outputs of a certain hidden layer (usually the last) based on the belief that the normal and anomalous inputs are significantly different in a certain space [86], [87]. The anomalous input can be adversarial input of evasion attack or an input with triggers for poisoning attack, such that the technology is defensive against backdoor poisoning attack and evasion attack and is validated for DDMs [87], [88], [89]. However, this technology does not work for DGMs.…”

Dual-filtering (DF) schemes for learning systems to prevent adversarial attacks

“…Metzen et al implemented deep neural networks with a small "detector" sub-network were trained on the binary classification task of distinguishing factual data from data containing adversarial perturbations [56]. The same year, Madry et al [55] [44]. A different notable strategy was taken by researchers Pang et al They used thresholding approach as the detector to filter out adversarial examples for reliable predictions [63].…”

A Unified Framework for Analyzing and Detecting Malicious Examples of DNN Models

“…The third direction is to introduce a preprocessing function to transform the input samples and remove the adversarial perturbations by gradient masking [3,10,18,42,56]. The last category is to detect adversarial examples [4,13,23,34,51,54,58]. Compared with the first three directions, these methods do not need to train a new model with different structures or datasets, or to alter the inference computing pipeline.…”

“…Detecting AEs. This methodology [23] explores the sample behaviors in the activation space of different network layers. The hypothesis is that the behaviors of normal samples are different from that of adversarial examples.…”