Fast is better than free: Revisiting adversarial training

Abstract

Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD). In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice. Specifically, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has significantly lower cost. Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efficient training of deep networks, allowing us to learn a robust CIFAR10 classifier with 45% robust accuracy to PGD attacks with = 8/255 in 6 minutes, and a robust ImageNet classifier with 43% robust accuracy at = 2/255 in 12 hours, in comparison to past work based on "free" adversarial training which took 10 and 50 hours to reach the same respective thresholds. Finally, we identify a failure mode referred to as "catastrophic overfitting" which may have caused previous attempts to use FGSM adversarial training to fail. All code for reproducing the experiments in this paper as well as pretrained model weights are at https://github.com/locuslab/fast_adversarial.

MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

“…We observed that addition of random augmentation further improves the results of our method. Our method outperforms both Free [67] and Fast [80] AT in accuracy and robustness, significantly.…”

“…For Table 1, we take clean accuracy and auto-attack results from [18] and PGD-100 results are the best PGD attack reported results (with the same or similar setting as ours) taken from the respective papers. For Table 2, we take Shafahi et al [67], Wong et al [80]'s reported results and evaluated our model with the same settings of PGD attack. For Table 3, we train the same models with PGD7-AT [50], RKD [26] and our method.…”

“…The adversarial training consists of an inner, iterative maximization loop to augment natural examples with adversarial perturbations, and an outer minimization loop similar to normal training. Many different methods have been introduced to improve robustness [77,15,53,92,80,67,95,79,7,82,40,45,57], but all of them are fundamentally based on the principle of training on adversarially augmented examples.…”

Robust and Privacy-Preserving Collaborative Learning: A Comprehensive Survey

“…Deep models are vulnerable to adversarial examples that are maliciously constructed to mislead the models to output wrong predictions but visually indistinguishable from normal samples [182]- [185]. Adversarial training [186]- [188] is one of the most effective approaches to defend deep models against adversarial examples and enhance their robustness. Its main idea is to augment training data with existing adversarial example generation methods during the training process.…”

“…For example, Shafahi et al [186] proposed an efficient adversarial training algorithm that recycles the gradient information computed at each iteration to eliminate the overhead cost of generating adversarial examples. Wong et al [188] propose to utilize Fast Gradient Sign Method (FGSM) [182] during the adversarial training process. They introduce random initialization points to improve the effectiveness the projected gradient descent based training.…”