A Unified Gradient Regularization Family for Adversarial Examples

Abstract

Abstract-Adversarial examples are augmented data points generated by imperceptible perturbation of input samples. They have recently drawn much attention with the machine learning and data mining community. Being difficult to distinguish from real examples, such adversarial examples could change the prediction of many of the best learning models including the state-of-the-art deep learning models. Recent attempts have been made to build robust models that take into account adversarial examples. However, these methods can either lead to performance drops or lack mathematical motivations. In this paper, we propose a unified framework to build robust machine learning models against adversarial examples. More specifically, using the unified framework, we develop a family of gradient regularization methods that effectively penalize the gradient of loss function w.r.t. inputs. Our proposed framework is appealing in that it offers a unified view to deal with adversarial examples. It incorporates another recently-proposed perturbation based approach as a special case. In addition, we present some visual effects that reveals semantic meaning in those perturbations, and thus support our regularization method and provide another explanation for generalizability of adversarial examples. By applying this technique to Maxout networks, we conduct a series of experiments and achieve encouraging results on two benchmark datasets. In particular,we attain the best accuracy on MNIST data (without data augmentation) and competitive performance on CIFAR-10 data.

Securing Connected & Autonomous Vehicles: Challenges Posed by Adversarial Machine Learning and the Way Forward

“…As a result, adversarial examples with small perturbations were unlikely to modify the output of deep models but this increases the training complexity with a factor of two. The notion of penalizing the gradient of loss function of models with respect to the inputs for robustification has been already been investigated in [191]. 4) Classifier Robustifying: In this method, classification models that are robust to adversarial attacks are designed from the ground up instead of detecting or transforming them.…”

Unifying Adversarial Training Algorithms with Data Gradient Regularization

“…, λ m are the weight coefficients of the terms in the DataGrad loss function. Close to our work, (Lyu et al, 2015) present a heuristic way to optimize a special case of this objective. By directly provding an algorithm, our analysis can explain what their algorithm optimizes.…”

“…In an approach that ends up closely related to ours, (Lyu et al, 2015) consider the objective min θ max r:||r||p≤σ L(x + r; θ) and a linearized inner version max r:||r||p≤σ L(x) + ∇ x L T r. They iteratively select r by optimizing the latter and θ by back-propagation on the former (with r fixed). Since the θ update is not directly minimizing the linearized objective, (Lyu et al, 2015) claimed the procedure was only an approximation of what we call the DataGrad objective. However, their method devolves to training on adversarial examples, so as before, Equation 4 shows they are actually optimizing the DataGrad objective but with r = 1 and λ 0 and λ 1 carefully chosen to eliminate the…”

“…Instead, they utilized approximations such as a shallow layerwise gradient penalty (Gu and Rigazio, 2014), also used for regularizing contractive autoencoders (Gu and Rigazio, 2014). Meanwhile (Lyu et al, 2015) also presented a heuristic algorithm for this objective.…”

Review of Artificial Intelligence Adversarial Attack and Defense Technologies

“…Biggio et al [8] used a regularization method to limit the vulnerability of data when training an SVM model. The works [47][48][49] used the regularization method to improve the robustness of the algorithm and achieved good results. The second method is defensive distillation [14], which produced a model with a smoother output surface and less sensitivity to disturbance, so as to improve the robustness of the model, and can reduce the success rate of adversarial attack by 90%.…”