Adversarial Robust Model Compression using In-Train Pruning

Vemparala, Manoj-Rohit; Fasfous, Nael; Frickenstein, Alexander; Sarkar, Sreetama; Zhao, Qi; Kühn, Sabine; Frickenstein, Lukas; Singh, Anupinder; Unger, Christian; Nagaraja, Naveen-Shankar; Wressnegger, Christian; Stechele, Walter

doi:10.1109/cvprw53098.2021.00016

Cited by 12 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior studies indicate that sparse models tend to underperform in Compression Identified Examples (CIE), suggesting that the pruning process exacerbates the inherent algorithmic biases hidden within the datasets (Hooker et al, 2020). In Computer Vision (CV), simultaneous optimization of model pruning and adversarial training has been advocated as an effective solution to this issue (Gui et al, 2019;Ye et al, 2019;Sehwag et al, 2020;Vemparala et al, 2021). In NLP, Du et al (2023) propose to prevent model overfitting on easy samples by leveraging sample difficulty in the context of pruning.…”

Section: Robust Model Pruningmentioning

confidence: 99%

Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models

Li,

Lei,

Cheng

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

The pruning objective has recently extended beyond accuracy and sparsity to robustness in language models. Despite this, existing methods struggle to enhance robustness against adversarial attacks when continually increasing model sparsity and require a retraining process. As humans step into the era of large language models, these issues become increasingly prominent. This paper proposes that the robustness of language models is proportional to the extent of pre-trained knowledge they encompass. Accordingly, we introduce a post-training pruning strategy designed to faithfully replicate the embedding space and feature space of dense language models, aiming to conserve more pretrained knowledge during the pruning process. In this setup, each layer's reconstruction error not only originates from itself but also includes cumulative error from preceding layers, followed by an adaptive rectification. Compared to other state-of-art baselines, our approach demonstrates a superior balance between accuracy, sparsity, robustness, and pruning cost with BERT on datasets SST2, IMDB, and AG-News, marking a significant stride towards robust pruning in language models.

show abstract

Section: Robust Model Pruningmentioning

confidence: 99%

Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models

Li,

Lei,

Cheng

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…To date several prior works have explored the efficient integration of model compression to the adversarial training process. To be specific, (Sehwag et al 2019;Ye et al 2019;Sehwag et al 2020;Vemparala et al 2021;Xie et al 2020a;Guo et al 2018;Madaan, Shin, and Hwang 2020;Rakin et al 2019;Özdenizci and Legenstein 2021) investigate the robustnessaware pruning to achieve high model sparsity and adversarial robustness. Also, (Fu et al 2021;Lin, Gan, and Han 2019) propose several approaches to develop low-bit precision robust models.…”

Section: Related Workmentioning

confidence: 99%

“…Motivated by this idea, several prior works have explored to examine and further improve the robustness of the compressed DNNs. In particular, because of the popularity of pruning in many model compression tasks, most of the existing compactness/robustness co-exploration efforts (Sehwag et al 2020(Sehwag et al , 2019Ye et al 2019;Vemparala et al 2021) focus on efficient approaches to generate the robust pruned DNN models. To be specific, (Ye et al 2019) demonstrates that the model robustness and compactness can co-exist for neural networks via concurrent adversarial training and weight pruning.…”

Section: Introductionmentioning

confidence: 99%

CSTAR: Towards Compact and Structured Deep Neural Networks with Adversarial Robustness

Phan

Yin

Yang

et al. 2023

AAAI

View full text Add to dashboard Cite

Model compression and model defense for deep neural networks (DNNs) have been extensively and individually studied. Considering the co-importance of model compactness and robustness in practical applications, several prior works have explored to improve the adversarial robustness of the sparse neural networks. However, the structured sparse models obtained by the existing works suffer severe performance degradation for both benign and robust accuracy, thereby causing a challenging dilemma between robustness and structuredness of compact DNNs. To address this problem, in this paper, we propose CSTAR, an efficient solution that simultaneously impose Compactness, high STructuredness and high Adversarial Robustness on the target DNN models. By formulating the structuredness and robustness requirement within the same framework, the compressed DNNs can simultaneously achieve high compression performance and strong adversarial robustness. Evaluations for various DNN models on different datasets demonstrate the effectiveness of CSTAR. Compared with the state-of-the-art robust structured pruning, CSTAR shows consistently better performance. For instance, when compressing ResNet-18 on CIFAR-10, CSTAR achieves up to 20.07% and 11.91% improvement for benign accuracy and robust accuracy, respectively. For compressing ResNet-18 with 16x compression ratio on Imagenet, CSTAR obtains 8.58% benign accuracy gain and 4.27% robust accuracy gain compared to the existing robust structured pruning.

show abstract

“…HAQ (Hardware-aware Quantization) used reinforcement learning to select bit-width for weights and activations to quantize a model during training while considering hardware constraints [46]. Vemparala et al [10] identified the optimal layers precision during the training process. They added hardware awareness to the training optimizer by using a differentiable Gaussian process regressor that provides accurate hardware predictions.…”

Section: Group A: Single-objective Hardware-aware Compressionmentioning

confidence: 99%