Better Robustness by More Coverage: Adversarial and Mixup Data Augmentation for Robust Finetuning

Abstract

Pretrained language models (PLMs) perform poorly under adversarial attacks. To improve the adversarial robustness, adversarial data augmentation (ADA) has been widely adopted to cover more search space of adversarial attacks by adding textual adversarial examples during training. However, the number of adversarial examples for text augmentation is still extremely insufficient due to the exponentially large attack search space. In this work, we propose a simple and effective method to cover a much larger proportion of the attack search space, called Adversarial and Mixup Data Augmentation (AMDA). Specifically, AMDA linearly interpolates the representations of pairs of training samples to form new virtual samples, which are more abundant and diverse than the discrete text adversarial examples in conventional ADA. Moreover, to fairly evaluate the robustness of different models, we adopt a challenging evaluation setup, which generates a new set of adversarial examples targeting each model. In text classification experiments of BERT and RoBERTa, AMDA achieves significant robustness gains under two strong adversarial attacks and alleviates the performance degradation of ADA on the clean data. Our code is available at: https://github.com/thunlp/MixADA.

Cyberbullying Classifiers are Sensitive to Model-Agnostic Perturbations

“…Most are discussed in Emmery et al (2021); however, some new work is specifically of interest to data augmentation, such as improving the substitutions using beam search (Zhao et al, 2021, as opposed to the simultaneous rollout we used in the current work). More broadly, adversarial training (Si et al, 2021;Pan et al, 2021), implementing more robust stylometric features (Markov et al, 2021), or modelbased weightings of the augmentation models could be explored; e.g., by selecting instances with a generation model in the loop (Anaby-Tavor et al, 2020). This could be a particularly worthwhile option when focusing on conversation scopes, rather than message-level cyberbullying content (Emmery et al, 2019).…”

NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

“…And the relevant keywords which categorise the transformation. t/evaluation data splits allows for testing the robustness of models and for identifying possible biases; on the other hand, applying transformations and filters to training data (data augmentation) allows for possibly mitigating the detected robustness and bias issues (Wang et al, 2021b;Pruksachatkun et al, 2021;Si et al, 2021).…”

“…For example, "Stillwater is not a 2010 American liveaction/animated dark fantasy adventure film" turns into "Stillwater !is film". Zhang et al (2021) used a similar idea to this transformation.…”

Data Augmentation Approaches in Natural Language Processing: A Survey

“…Given the original samples, Cheng et al [25] firstly construct their adversarial samples following [75], and then apply two Mixup strategies named P adv and P aut : The former interpolates between adversarial samples, and the latter interpolates between the two corresponding original samples. Similarly, Sun et al [76], Bari et al [77] , and Si et al [78] both apply such Mixup method for text classification. Sun et al [76] propose Mixup-Transformer which combines Mixup with transformer-based pretrained architecture.…”