Consistency Training with Virtual Adversarial Discrete Perturbation

Abstract

We propose an effective consistency training framework that enforces a training model's predictions given original and perturbed inputs to be similar by adding a discrete noise that would incur the highest divergence between predictions. This virtual adversarial discrete noise obtained by replacing a small portion of tokens while keeping original semantics as much as possible efficiently pushes a training model's decision boundary. Moreover, we perform an iterative refinement process to alleviate the degraded fluency of the perturbed sentence due to the conditional independence assumption. Experimental results show that our proposed method outperforms other consistency training baselines with text editing, paraphrasing, or a continuous noise on semisupervised text classification tasks and a robustness benchmark 1 .

Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering

“…However, the consistency training framework is also applicable when only the labeled samples are available (Miyato et al, 2018;Jiang et al, 2019;Asai and Hajishirzi, 2020). The consistency regularization requires adding noise to the sample, which can be either discrete (Xie et al, 2020;Asai and Hajishirzi, 2020;Park et al, 2021) or continuous (Miyato et al, 2016;Jiang et al, 2019). Existing works regularize the predictions of the perturbed samples to be equivalent to be that of the originals'.…”