AdvRush: Searching for Adversarially Robust Neural Architectures

Abstract

Deep neural networks continue to awe the world with their remarkable performance. Their predictions, however, are prone to be corrupted by adversarial examples that are imperceptible to humans. Current efforts to improve the robustness of neural networks against adversarial examples are focused on developing robust training methods, which update the weights of a neural network in a more robust direction. In this work, we take a step beyond training of the weight parameters and consider the problem of designing an adversarially robust neural architecture with high intrinsic robustness. We propose AdvRush, a novel adversarial robustness-aware neural architecture search algorithm, based upon a finding that independent of the training method, the intrinsic robustness of a neural network can be represented with the smoothness of its input loss landscape. Through a regularizer that favors a candidate architecture with a smoother input loss landscape, AdvRush successfully discovers an adversarially robust neural architecture. Along with a comprehensive theoretical motivation for Ad-vRush, we conduct an extensive amount of experiments to demonstrate the efficacy of AdvRush on various benchmark datasets. Notably, on CIFAR-10, AdvRush achieves 55.91% robust accuracy under FGSM attack after standard training and 50.04% robust accuracy under AutoAttack after 7-step PGD adversarial training.

$A^{3}D$: A Platform of Searching for Robust Neural Architectures and Efficient Adversarial Attacks

“…The non-differentiable methods adopt the robust accuracy of FGSM or PGD adversarial training as the evaluation, which is relatively time-consuming Guo et al (2020); Liu and Jin (2021b); Vargas et al (2019). Further, the differentiable methods take the Jacobian or Hessian matrix to measure the robustness of the models, resulting in the great acceleration of the search procedure Mok et al (2021); Hosseini et al (2021).…”

“…Apart from the three above game-based robustness evaluation metrics, we also include the quantified metric such as the Frobenius (F ) norms of Jacobian matrix of the input data, which does not rely on adversarial noise generated by various adversarial attacks, resulting in an easy combination with differentiable neural architecture search methods Hosseini et al (2021); Mok et al (2021).…”

“…where J denotes the Jacobian matrix of the output vector of 10 classifier with respect to input data, i represents the number of input sample. Similarly, the calculation of the F norms of Hessian matrix can be seen in Mok et al (2021).…”

“…Hosseini et al Hosseini et al (2021) incorporate two robustness metrics, including Jacobian matrix and certified lower bound into partial channel connections for memory-efficient architecture search (PC-DARTS) Xu et al (2019). Similarly, Mok et al Mok et al (2021) use the Hessian matrix information of input data as the robustness evaluation to guide the process of DARTS Liu et al (2018).…”

“…The reason is that the metrics with adversarial training have been proved to bring overestimation of the robustness, resulting in the searched model by NAS not being robust enough. For example, the robust accuracy of five searched model architectures is presented in Fig 1, which can be obtained from the report of AdvRush Mok et al (2021). If the evaluation just considers the robust accuracy of FGSM or PGD, it could cause the misjudgment of robust architectures in NAS.…”