Training Lightweight yet Competent Network via Transferring Complementary Features

Zhang, Xiaobing; Lu, Shijian; Gong, Haigang; Liu, Minghui; Liu, Ming

doi:10.1007/978-3-030-63820-7_65

Cited by 7 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To alleviate AT's time complexity, researchers attempted to attain state-of-the-art robustness with single-step attacks (Zhang et al, 2019a;Shafahi et al, 2019;Wong et al, 2020) by adopting accumulative perturbation and perturbation initialization. Shafahi et al (2019) employed a single backpropagation step to update the model weights and generate the adversarial perturbations.…”

Section: Literature Reviewmentioning

confidence: 99%

Perturbation Augmentation for Adversarial Training with Diverse Attacks

Serbes,

Baytaş

2024

Gazi University Journal of Science Part A: Engineering and Innovation

View full text Add to dashboard Cite

Adversarial Training (AT) aims to alleviate the vulnerability of deep neural networks to adversarial perturbations. However, the AT techniques struggle to maintain the performance on natural samples while improving the deep model’s robustness. The absence of perturbation diversity in generated during the adversarial training degrades the generalizability of the robust models, causing overfitting to particular perturbations and a decrease in natural performance. This study proposes an adversarial training framework that augments adversarial directions from a single-step attack to address the trade-off between robustness and generalization. Inspired by feature scattering adversarial training, the proposed framework computes a principal adversarial direction with a single-step attack that finds a perturbation disrupting the inter-sample relationships in the mini-batch during adversarial training. The principal direction obtained at each iteration is augmented by sampling new adversarial directions within a region spanning 45 degrees from the principal adversarial direction. The proposed adversarial training approach does not require extra backpropagation steps in adversarial direction augmentation. Therefore, generalization of the robust model is improved without posing an additional burden on the feature scattering adversarial training. Experiments on CIFAR-10, CIFAR-100, SVHN, Tiny-ImageNet, and The German Traffic Sign Recognition Benchmark consistently improve the accuracy on adversarial with an almost pristine natural performance.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Perturbation Augmentation for Adversarial Training with Diverse Attacks

Serbes,

Baytaş

2024

Gazi University Journal of Science Part A: Engineering and Innovation

View full text Add to dashboard Cite

show abstract

“…Despite the popularity of adversarial training to defend models, it has a high cost of generating strong adversarial examples, Free Adversarial Training (Free-AT) [92] uses the gradient information when updating model parameters to generate the adversarial examples, eliminating the previously mentioned overhead. Another optimization regarding the computational costs, consists in only considering the first layer of a network for forward and backpropagation, effectively reducing the amount of propagation to one in each update, named as You Only Propagate Once (YOPO) [93], due to the adversary update being only related to the first layer. Exploring the different layers of a network, Latent Adversarial Training (LAT) [94] consists of finetuning adversarial-trained models to ensure robustness at the latent level, because the latent layer is significantly vulnerable to adversarial perturbations of small magnitude.…”

Section: A Adversarial Trainingmentioning

confidence: 99%

How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses

Costa,

Roxo,

Proença

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Deep Learning is currently used to perform multiple tasks, such as object recognition, face recognition, and natural language processing. However, Deep Neural Networks (DNNs) are vulnerable to perturbations that alter the network prediction, named adversarial examples, which raise concerns regarding the usage of DNNs in critical areas, such as Self-driving Vehicles, Malware Detection, and Healthcare. This paper compiles the most recent adversarial attacks in Object Recognition, grouped by the attacker capacity and knowledge, and modern defenses clustered by protection strategies, providing background details to understand the topic of adversarial attacks and defenses. The new advances regarding Vision Transformers are also presented, which have not been previously done in the literature, showing the resemblance and dissimilarity between this architecture and Convolutional Neural Networks. Furthermore, the most used datasets and metrics in adversarial settings are summarized, along with datasets requiring further evaluation, which is another contribution. This survey compares the state-of-the-art results under different attacks for multiple architectures and compiles all the adversarial attacks and defenses with available code, comprising significant contributions to the literature. Finally, practical applications are discussed, and open issues are identified, being a reference for future works.INDEX TERMS Adversarial attacks, adversarial defenses, datasets, evaluation metrics, review, vision transformers.

show abstract

“…Free AT [26] updates network parameters while generating AEs at the same time. YOPO [27] restricts most of the forward and back propagation within the first layer of the network during AE updates. Fast AT [7] replace multi-step attacks with FGSM and generate AEs with single-step gradient.…”

Section: B Efficient Adversarial Trainingmentioning

confidence: 99%

“…Since the network interacts directly with the perturbation via the first layer, we concentrate our analysis on the first layer, as done in [27]. To further study the non-linear characteristics of the network, the features of all training data after the first activation layer are extracted.…”

Section: Self-fitting Phenomenon In Fatmentioning

confidence: 99%

Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective

He¹,

Li²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Although fast adversarial training provides an efficient approach for building robust networks, it may suffer from a serious problem known as catastrophic overfitting (CO), where the multi-step robust accuracy suddenly collapses to zero. In this paper, we for the first time decouple the FGSM examples into data-information and self-information, which reveals an interesting phenomenon called "self-fitting". Self-fitting, i.e., DNNs learn the self-information embedded in single-step perturbations, naturally leads to the occurrence of CO. When self-fitting occurs, the network experiences an obvious "channel differentiation" phenomenon that some convolution channels accounting for recognizing self-information become dominant, while others for data-information are suppressed. In this way, the network learns to only recognize images with sufficient self-information and loses generalization ability to other types of data. Based on self-fitting, we provide new insight into the existing methods to mitigate CO and extend CO to multi-step adversarial training. Our findings reveal a self-learning mechanism in adversarial training and open up new perspectives for suppressing different kinds of information to mitigate CO.Impact Statement-Fast adversarial training is an effective and efficient adversarial training method. However, it is prone to instability and can lead to catastrophic overfitting (CO). In this paper, we have revealed for the first time the existence of model self-information in adversarial examples, and argue that fitting self-information (self-fitting) is one of the factors that contribute to CO. Our findings can further aid in the understanding of CO in fast adversarial training and even multi-step adversarial training, inspiring the generation of more stable and efficient adversarial training algorithms. The discovery of self-fitting is not only for adversarial attack but also helpful other methods that involved network information, like curriculum learning, active learning, and self-supervised learning.

show abstract

Training Lightweight yet Competent Network via Transferring Complementary Features

Cited by 7 publications

References 9 publications

Perturbation Augmentation for Adversarial Training with Diverse Attacks

Perturbation Augmentation for Adversarial Training with Diverse Attacks

How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses

Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective

Contact Info

Product

Resources

About