Consistency Training with Virtual Adversarial Discrete Perturbation

Abstract

Consistency training regularizes a model by enforcing predictions of original and perturbed inputs to be similar. Previous studies have proposed various augmentation methods for the perturbation but are limited in that they are agnostic to the training model. Thus, the perturbed samples may not aid in regularization due to their ease of classification from the model. In this context, we propose an augmentation method of adding a discrete noise that would incur the highest divergence between predictions. This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model's decision boundary. Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semi-supervised text classification tasks and a robustness benchmark 1 .

Improving the Sample Efficiency of Prompt Tuning with Domain Adaptation

“…After that, the neural network minimizes cross-entropy on labeled data and the KL-divergence under disturbance on all data. Similar ideas have been explored in (Cicek and Soatto, 2019;Kim et al, 2019;Park et al, 2022).…”

“…Consistency training methods (Laine and Aila, 2017;Sajjadi et al, 2016;Wei and Zou, 2019;Ng et al, 2020;Xie et al, 2020) force the model to make consistent predictions against small perturbations. For example, Park et al (2022) produce discrete virtual adversarial noise to the token embeddings. (Zhou et al, 2019) and unsupervised domain adaptation (UDA) (Wang et al, 2019;Long et al, 2022), depending on whether the target-domain data are labeled or unlabeled.…”

On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey

“…This kind of regularization technique has been widely adopted in NLP. For example, [59] produce discrete virtual adversarial noise to the token embeddings. [121] apply mixup to perturb the spans of the input texts for text classification for consistency training.…”

Regularising Fisher Information Improves Cross-lingual Generalisation

“…Many recent works use 'consistency regularisation' to improve the generalisation of fine-tuned pre-trained models, both multilingual and Englishonly (Jiang et al, 2020;Aghajanyan et al, 2020;Zheng et al, 2021;Park et al, 2021;Liang et al, 2021). These works encourage model outputs to be similar between a perturbed and normal version of the input, usually via penalising the Kullback-Leibler (KL) divergence between the probability distribution of the perturbed and normal model.…”

“…They also rarely compare to traditional regularisation methods like dropout or L2 regularisation. Finally such methods either require a complex adversarial training step (Jiang et al, 2020;Park et al, 2021), or tuning many hyper-parameters like type of noise, level of noise, and weight given to the consistency loss term.…”