RobustART: Benchmarking Robustness on Architecture Design and Training Techniques

Abstract

Deep neural networks (DNNs) are vulnerable to adversarial noises, which motivates the benchmark of model robustness. Existing benchmarks mainly focus on evaluating the defenses, but there are no comprehensive studies of how architecture design and general training techniques affect robustness. Comprehensively benchmarking their relationships will be highly beneficial for better understanding and developing robust DNNs. Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet (including open-source toolkit, pre-trained model zoo, datasets, and analyses) regarding ARchitecture design (44 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ general techniques, e.g., data augmentation) towards diverse noises (adversarial, natural, and system noises). Extensive experiments revealed and substantiated several insights for the first time, for example: (1) adversarial training largely improves the clean accuracy and all types of robustness for Transformers and MLP-Mixers; (2) with comparable sizes, CNNs > Transformers > MLP-Mixers on robustness against natural and system noises; Transformers > MLP-Mixers > CNNs on adversarial robustness;(3) for some light-weight architectures (e.g., EfficientNet, MobileNetV2, and Mo-bileNetV3), increasing model sizes or using extra training data cannot improve robustness. Our benchmark http://robust.art/: (1) presents an open-source platform for conducting comprehensive evaluation on diverse robustness types;(2) provides a variety of pre-trained models with different training techniques to facilitate robustness evaluation; (3) proposes a new view to better understand the mechanism towards designing robust DNN architectures, backed up by the analysis. We will continuously contribute to building this ecosystem for the community.

Robust Unlearnable Examples: Protecting Data Against Adversarial Learning

“…Other advances include TRADE , FAT , GAIRAT (Zhang et al, 2021), (Song et al, 2020), Jiang et al (2021), Stutz et al (2020), Singla & Feizi (2020), Wu et al (2020a), and. Tang et al (2021) propose an adversarial robustness benchmark regarding architecture design and training techniques. Some works also attempt to interpret how machine learning models gain robustness (Ilyas et al, 2019;Zhang & Zhu, 2019).…”

$A^{3}D$: A Platform of Searching for Robust Neural Architectures and Efficient Adversarial Attacks

“…System noises refer to the inaccuracy inherent to a system due to the inconsistent implementations of decoding and resize, such as ImageNet-S Wang et al (2021b); Tang et al (2021). The system noise adapted for the CIFAR10 dataset is created in a similar way in our framework.…”

“…On the one hand, the robustness evaluation is not comprehensive. Current work evaluate the robustness under different adversarial attack and quantified metrics, but the robustness of the model also needs to consider the robust accuracy under other types of noises such as natural noise Hendrycks et al (2021a) and system noise Wang et al (2021a); Tang et al (2021). On the other hand, under adversarial noise, the robustness evaluation is not reliable.…”

Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks

“…Another important observation is that using WRNs instead of ResNets (RNs) can bring ∼ 3%-5% more robustness [16][17][18]. Other works also suggest that adversarial training requires deeper and wider models [15,26,55]. Also, the skip-connection operation used in WRN has been found can improve robustness for deeper architectures [56].…”