Improving Adversarial Robustness by Enforcing Local and Global Compactness

Abstract

The fact that deep neural networks are susceptible to crafted perturbations severely impacts the use of deep learning in certain domains of application. Among many developed defense models against such attacks, adversarial training emerges as the most successful method that consistently resists a wide range of attacks. In this work, based on an observation from a previous study that the representations of a clean data example and its adversarial examples become more divergent in higher layers of a deep neural net, we propose the Adversary Divergence Reduction Network which enforces local/global compactness and the clustering assumption over an intermediate layer of a deep neural network. We conduct comprehensive experiments to understand the isolating behavior of each component (i.e., local/global compactness and the clustering assumption) and compare our proposed model with state-of-the-art adversarial training methods. The experimental results demonstrate that augmenting adversarial training with our proposed components can further improve the robustness of the network, leading to higher unperturbed and adversarial predictive performances.

Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning

“…The experiments are conducted on the CIFAR10 and CIFAR100 datasets. The comparison on Tables (4,5,6) shows that our ASCL method significantly improves both adversarial training based models by around 4% to 5% robust accuracy. Moreover, our ASCL also outperforms the ADR method by around 1% to 2% with ResNet20 and 2.6% with ResNet50.…”

“…We apply the two versions, ASCL and Leaked-ASCL, as a regularization on top of two adversarial training methods, PGD adversarial training (ADV) [22] and TRADES [36]. We compare our methods with ADR which is the state-ofthe-art regularization technique as proposed in [4]. The experiments are conducted on the CIFAR10 and CIFAR100 datasets.…”

“…Among them, the adversarial training methods (e.g, FGSM, PGD adversarial training [13,22] and TRADES [36] that utilize adversarial examples as training data, have been one of the most effective approaches, which truly boost the model robustness without the facing the problem of obfuscated gradients [3]. In adversarial training, recent works [34,4] show that reducing the divergence of the representations of images and their adversarial examples in latent space (e.g., the feature space output from an intermediate layer of a classifier) can significantly improve the robustness. For example, in [4], latent representations of images in the same class are pulled closer together than those in different classes, which led to a more compact latent space and consequently, better robustness.…”

“…In adversarial training, recent works [34,4] show that reducing the divergence of the representations of images and their adversarial examples in latent space (e.g., the feature space output from an intermediate layer of a classifier) can significantly improve the robustness. For example, in [4], latent representations of images in the same class are pulled closer together than those in different classes, which led to a more compact latent space and consequently, better robustness.…”

Improving Ensemble Robustness by Collaboratively Promoting and Demoting Adversarial Robustness

“…Many ADV's variants have been developed including but not limited to: (1) difference in the choice of adversarial examples, e.g., the worst-case examples (I. J. Goodfellow, Shlens, and Szegedy, 2015) or most divergent examples (Hongyang Zhang et al, 2019), (2) difference in the searching of adversarial examples, e.g., non-iterative FGSM, Rand FGSM with random initial point or PGD with multiple iterative gradient descent steps (Madry et al, 2018;Shafahi et al, 2019), (3) difference in additional regularizations, e.g., adding constraints in the latent space (Haichao Zhang and Wang, 2019;Bui et al, 2020), (4) difference in model architecture, e.g., activation function (Xie et al, 2020) or ensemble models (Pang, Xu, et al, 2019).…”