Unlabeled Data Improves Adversarial Robustness

Abstract

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. [35] that shows a sample complexity gap between standard and robust classification. We prove that this gap does not pertain to labels: a simple semisupervised learning procedure (self-training) achieves robust accuracy using the same number of labels required for standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) ∞ robustness against several strong attacks via adversarial training and (ii) certified 2 and ∞ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels as well.

MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

“…Please see Appendix.B for more details and complete proof. Here the distillation loss is formulated by the prediction accuracy for pseudo-labels generated by a learnt teacher model, which is also considered by [15].…”

“…The adversarial training consists of an inner, iterative maximization loop to augment natural examples with adversarial perturbations, and an outer minimization loop similar to normal training. Many different methods have been introduced to improve robustness [77,15,53,92,80,67,95,79,7,82,40,45,57], but all of them are fundamentally based on the principle of training on adversarially augmented examples.…”

“…Even over-parameterized models that can easily fit data for normal training [93] may have insufficient capacity to fit adversarially augmented data [98]. The sample complexity of adversarial training can be significantly higher than normal training, and it requires more labeled [65] or unlabeled data [77,15,53,92]. The inner maximization for the generation of adversarial perturbations for adversarial training is significantly more expensive computationally as it requires iterative gradient steps with respect to the inputs (e.g., adversarial training takes ∼7x more time compared with normal training).…”

Improving Robustness with Image Filtering

“…For this reason, significant efforts have been made to improve the robustness of models without resorting to AT. Proposed solutions cover a wide range of techniques based on curvature regularization [17], robust optimization to improve local stability [23], the use of additional unlabeled data [4], local linearization [21], Parseval networks [5], defensive distillation [19], model ensembles [18], channel-wise activations suppressing [2], feature denoising [29], self-supervised learning for adversarial purification [25], and input manipulations [7,9,15]. All the listed techniques, except those based on input manipulations, require training the model or an auxiliary module from scratch.…”

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

“…for both supervised classifier and attack generator (that ensures misclassification). The recent work [22][23][24] demonstrated that with a properly-designed attacker's objective, AT-type defenses can be generalized to the semi-supervised setting, and showed that the incorporation of additional unlabeled data could further improve adversarial robustness in image classification. Such an extension from supervised to semi-supervised defenses further inspires us to ask whether there exist unsupervised defenses that can eliminate the prerequisite of labeled data but improve model robustness.…”