RobustBench: a standardized adversarial robustness benchmark

Abstract

Evaluation of adversarial robustness is often error-prone leading to overestimation of the true robustness of models. While adaptive attacks designed for a particular defense are a way out of this, there are only approximate guidelines on how to perform them. Moreover, adaptive evaluations are highly customized for particular models, which makes it difficult to compare different defenses. Our goal is to establish a standardized benchmark of adversarial robustness, which as accurately as possible reflects the robustness of the considered models within a reasonable computational budget. This requires to impose some restrictions on the admitted models to rule out defenses that only make gradient-based attacks ineffective without improving actual robustness. We evaluate robustness of models for our benchmark with AutoAttack, an ensemble of white-and black-box attacks which was recently shown in a large-scale study to improve almost all robustness evaluations compared to the original publications. Our leaderboard, hosted at http://robustbench.github.io/, aims at reflecting the current state of the art on a set of well-defined tasks in ∞ -and 2 -threat models with possible extensions in the future. Additionally, we open-source the library http://github.com/RobustBench/robustbench that provides unified access to state-of-the-art robust models to facilitate their downstream applications. Finally, based on the collected models, we analyze general trends in p -robustness and its impact on other tasks such as robustness to various distribution shifts and out-of-distribution detection.

What You See is Not What the Network Infers: Detecting Adversarial Examples Based on Semantic Contradiction

“…Therefore, we follow the white-box setting demonstrated in Sec. V-A, and use AEs generated by AutoAttack on each ATC from [18] for evaluation. As shown in Tab.…”

“…Dataset. All experiments are conducted on CIFAR-10, which serves as a standard task by several public robustness testbenches [18], [48].…”

“…Incorporating with Adversarial-Trained Classifier (ATC) can further improve the robustness performance of Con-traNet. We directly use an ATC, Gowal2020Uncovering [27], from RobustBench [18], to incorporate with ContraNet forming a robust classifier (ATC + ContraNet). Fig.…”

“…[7]. ATC we use here is Gowal2020Uncovering [27] from RobustBench [18] Robust Accuracy against Orthogonal-PGD. As shown in Tab.…”

“…V-C and VI and Re-buffi2021Fixing_70 in Sec. V-C, from [18]. † used by Rebuffi2021Fixing_28 and Sehwag2020Hydra in Sec.…”