Relating Adversarially Robust Generalization to Flat Minima

Stutz, David; Hein, Matthias; Schiele, Bernt

doi:10.1109/iccv48922.2021.00771

Cited by 25 publications

(21 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adversarial training has also been applied to corruption robustness [28,37] or to achieve robustness against multiple different attacks [39,56,60]. While [34,52] find particularly robust subsets and [57,65] employ adversarial training with weight perturbations, the poor robustness of subsets has not been addressed yet. However, we believe that subnets with poor robustness hamper the overall adversarial robustness.…”

Section: Related Workmentioning

confidence: 99%

Improving Robustness by Enhancing Weak Subnets

Guo¹,

Stutz²,

Schiele³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks have achieved great success in many computer vision tasks. However, deep networks have been shown to be very susceptible to corrupted or adversarial images, which often result in significant performance drops. In this paper, we observe that weak subnetwork (subnet) performance is correlated with a lack of robustness against corruptions and adversarial attacks. Based on that observation, we propose a novel robust training method which explicitly identifies and enhances weak subnets (EWS) during training to improve robustness. Specifically, we develop a search algorithm to find particularly weak subnets and propose to explicitly strengthen them via knowledge distillation from the full network. We show that our EWS greatly improves the robustness against corrupted images as well as the accuracy on clean data. Being complementary to many state-of-the-art data augmentation approaches, EWS consistently improves corruption robustness on top of many of these approaches. Moreover, EWS is also able to boost the adversarial robustness when combined with popular adversarial training methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Improving Robustness by Enhancing Weak Subnets

Guo¹,

Stutz²,

Schiele³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Wu et al [48] empirically verified that a flatter weight loss landscape often leads to a smaller robust generalization gap in AT. Stutz et al [40] studied the relationship between robust generalization and flatness of the robust loss landscape in the weight space. However, the above analysis and defense methods are either not general or effective for improving the robust generalization.…”

Section: Robust Overfitting In Atmentioning

confidence: 99%

“…Dong et al [12] hypothesized that robust overfitting comes from the memorization effect of the model on one-hot labels: as the one-hot labels of some samples are noisy, the model will remember those "hard" samples with noisy labels, leading to a decrease in robustness. Some works studied robust overfitting with weight loss landscape [48,40]. However, there is still a lack of general understanding about the overfitting issue, and more effective mitigation solutions are urgently needed.…”

Section: Introductionmentioning

confidence: 99%

Alleviating Robust Overfitting of Adversarial Training With Consistency Regularization

Zhang¹,

Gao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Adversarial training (AT) has proven to be one of the most effective ways to defend Deep Neural Networks (DNNs) against adversarial attacks. However, the phenomenon of robust overfitting, i.e., the robustness will drop sharply at a certain stage, always exists during AT. It is of great importance to decrease this robust generalization gap in order to obtain a robust model. In this paper, we present an in-depth study towards the robust overfitting from a new angle. We observe that consistency regularization, a popular technique in semi-supervised learning, has a similar goal as AT and can be used to alleviate robust overfitting. We empirically validate this observation, and find a majority of prior solutions have implicit connections to consistency regularization. Motivated by this, we introduce a new AT solution, which integrates the consistency regularization and Mean Teacher (MT) strategy into AT. Specifically, we introduce a teacher model, coming from the average weights of the student models over the training steps. Then we design a consistency loss function to make the prediction distribution of the student models over adversarial examples consistent with that of the teacher model over clean samples. Experiments show that our proposed method can effectively alleviate robust overfitting and improve the robustness of DNN models against common adversarial attacks.

show abstract

“…Recent works [5,9,65,71,75,90] suggest a promising alternative by integrating worst-case adversarial perturbations as data augmentations (i.e., AT). AT restricts the change of loss when its input is perturbed, leading to flattening the loss landscape [30,58]. As a result, the trained network's intrinsic feature manifold and loss landscape become smoother.…”

Section: Regularize Nerf With Robust Augmentationsmentioning

confidence: 99%

Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations

Chen¹,

Peihao²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural Radiance Field (NeRF) regresses a neural parameterized scene by differentially rendering multi-view images with ground-truth supervision. However, when interpolating novel views, NeRF often yields inconsistent and visually non-smooth geometric results, which we consider as a generalization gap between seen and unseen views. Recent advances in convolutional neural networks have demonstrated the promise of advanced robust data augmentations, either random or learned, in enhancing both in-distribution and out-of-distribution generalization. Inspired by that, we propose Augmented NeRF (Aug-NeRF), which for the first time brings the power of robust data augmentations into regularizing the NeRF training. Particularly, our proposal learns to seamlessly blend worst-case perturbations into three distinct levels of the NeRF pipeline with physical grounds, including (1) the input coordinates, to simulate imprecise camera parameters at image capture; (2) intermediate features, to smoothen the intrinsic feature manifold; and (3) pre-rendering output, to account for the potential degradation factors in the multi-view image supervision. Extensive results demonstrate that Aug-NeRF effectively boosts NeRF performance in both novel view synthesis (up to 1.5dB PSNR gain) and underlying geometry reconstruction. Furthermore, thanks to the implicit smooth prior injected by the triple-level augmentations, Aug-NeRF can even recover scenes from heavily corrupted images, a highly challenging setting untackled before. Our codes are available in https://github.com/VITA-Group/Aug-NeRF.

show abstract

Relating Adversarially Robust Generalization to Flat Minima

Cited by 25 publications

References 12 publications

Improving Robustness by Enhancing Weak Subnets

Improving Robustness by Enhancing Weak Subnets

Alleviating Robust Overfitting of Adversarial Training With Consistency Regularization

Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations

Contact Info

Product

Resources

About