Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data

Abstract

Self-training algorithms, which train a model to fit pseudolabels predicted by another previously-learned model, have been very successful for learning with unlabeled data using neural networks. However, the current theoretical understanding of self-training only applies to linear models. This work provides a unified theoretical analysis of selftraining with deep networks for semi-supervised learning, unsupervised domain adaptation, and unsupervised learning. At the core of our analysis is a simple but realistic "expansion" assumption, which states that a low-probability subset of the data must expand to a neighborhood with large probability relative to the subset. We also assume that neighborhoods of examples in different classes have minimal overlap. We prove that under these assumptions, the minimizers of population objectives based on self-training and input-consistency regularization will achieve high accuracy with respect to ground-truth labels. By using off-the-shelf generalization bounds, we immediately convert this result to sample complexity guarantees for neural nets that are polynomial in the margin and Lipschitzness. Our results help explain the empirical successes of recently proposed self-training algorithms which use input consistency regularization.

Federated Learning of Molecular Properties with Graph Neural Networks in a Heterogeneous Setting

“…An alternative way to define φ(•) for sample x l is the discrepancy value of the model's prediction between the sample and its adversarial neighbor (Wei et al 2020). Adversarial neighbor is a sample which is similar to x l in terms of the input graph g l but has the most different prediction.…”

Enhancing Counterfactual Classification via Self-Training

“…gorithm and show its convergence under proper initialization. Recent theoretical analysis (Wei et al 2020) and empirical evidence shows input consistency loss such as VAT loss (Miyato et al 2018) can further improve pseudolabeling in semi-supervised learning. Han et al (2019) points out that pseudolabel imputation can be viewed as minimizing minentropy as a type of Rényi entropy 1 1−α log( n i=1 p α i ) when α → ∞, and Shannon entropy in (Grandvalet and Bengio 2005) is the case when α → 1.…”

“…However, it has been shown that such a procedure may suffer from local minima (Grandvalet and Bengio 2005) or over-confident wrong pseudolabels (Zou et al 2019). Wei et al (2020) shows when the underlying data distribution and pseudolabeler satisfies expansion assumption (See Definition 3.1 and Assumption 4.1, 3.3 in Wei et al (2020)), self-training algorithms with input-consistency are able to achieve improvement from pseudolabeling (Theorem 4.3 in Wei et al (2020)). Intuitively, the condition states that there needs to be many correct neighbors around the errors made by pseudolabeler so that correct labels can refine the decision boundary.…”

“…Last, the separation assumption is automatically met given the definition of T . The proof is completed by directly applying Theorem 4.3 in Wei et al (2020). Here L(f ) is a weighted sum of input consistency loss and the loss for fitting pseudolabels, which offers the theoretical guarantee for CST.…”

“…Here L(f ) is a weighted sum of input consistency loss and the loss for fitting pseudolabels, which offers the theoretical guarantee for CST. In CST, the source loss is added for training stability, which is often used in self-training algorithms (Zou et al 2019;Wei et al 2020;Rizve et al 2021). For Equation 13, it is a direct result of sum of geometric sequence.…”