Unsupervised Data Augmentation for Consistency Training

Abstract

Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning. By substituting simple noising operations with advanced data augmentation methods, our method brings substantial improvements across six language and three vision tasks under the same consistency training framework. On the IMDb text classification dataset, with only 20 labeled examples, our method achieves an error rate of 4.20, outperforming the state-of-the-art model trained on 25,000 labeled examples. On a standard semi-supervised learning benchmark, CIFAR-10, our method outperforms all previous approaches and achieves an error rate of 2.7% with only 4,000 examples, nearly matching the performance of models trained on 50,000 labeled examples. Our method also combines well with transfer learning, e.g., when finetuning from BERT, and yields improvements in high-data regime, such as ImageNet, whether when there is only 10% labeled data or when a full labeled set with 1.3M extra unlabeled examples is used. 1

PLATINUM: Semi-Supervised Model Agnostic Meta-Learning using Submodular Mutual Information

“…Virtual Adversarial Training (VAT) (Miyato et al, 2018) uses an effective regularization technique that uses slight perturbations such that the prediction of the unlabeled samples is affected the most. More recent techniques like FixMatch (Sohn et al, 2020), MixMatch (Berthelot et al, 2019) and UDA (Xie et al, 2019) use data augmentations like flip, rotation, and crops to predict pseudo-labels. In this paper, we propose a new SSL technique that uses class-wise instantiations of SMI functions that mitigates the issue of class-imbalance in selected subsets and is comparatively robust to OOD classes in the unlabeled set.…”

Semantic-based Data Augmentation for Math Word Problems

“…[25] systematically examined some basic augmentation methods including random synonyms replacement, word insertion, etc. [27] utilized tf-idf to help determine which words to replace. [23] adopted k-nearest neighbors to find synonyms in word embedding space.…”

“…Neural MWP solvers can benefit from our augmentation strategies in terms of generalization and the ability of dealing with tiny local variances. Unlike other popular augmentation approaches [25,27,30], which may cause inconsistency of the questions and equations in MWP task, our augmentation methods are carefully designed for MWP task to ensure consistency.…”

SCR: Training Graph Neural Networks with Consistency Regularization

“…Temporal Ensembling [19] uses previous model checkpoints while Mean Teacher [26] uses an exponential moving average of model parameters. UDA [30] and ReMix-Match [18] sharpen the soft label to make the model to produce high-confidence predictions. UDA further reinforces consistency only when the highest probability of the predicted category distribution for soft labels is above a threshold.…”

“…However, the predictions made by the teacher could be noisy, especially at the beginning of the training process, which hinders the model from fitting to the supervised loss. A solution is to filter out low-quality predictions with the confidence-based masking strategy [30]. Specifically, we maintain a confident node set V 𝐶 ⊆ V 𝑈 in the training stage whose elements are unlabeled node with highly skewed predictions, i.e.,…”