Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning

Abstract

In this paper we revisit the idea of pseudo-labeling in the context of semi-supervised learning where a learning algorithm has access to a small set of labeled samples and a large set of unlabeled samples. Pseudo-labeling works by applying pseudo-labels to samples in the unlabeled set by using a model trained on combination of the labeled samples and any previously pseudo-labeled samples, and iteratively repeating this process in a self-training cycle. Current methods seem to have abandoned this approach in favor of consistency regularization methods that train models under a combination of different styles of self-supervised losses on the unlabeled samples and standard supervised losses on the labeled samples. We empirically demonstrate that pseudo-labeling can in fact be competitive with the state-of-the-art, while being more resilient to out-of-distribution samples in the unlabeled set. We identify two key factors that allow pseudo-labeling to achieve such remarkable results (1) applying curriculum learning principles and (2) avoiding concept drift by restarting model parameters before each self-training cycle. We obtain 94.91% accuracy on CIFAR-10 using only 4,000 labeled samples, and 68.87% top-1 accuracy on Imagenet-ILSVRC using only 10% of the labeled samples.

MaxMatch: Semi-Supervised Learning With Worst-Case Consistency

“…[34], [35], [36] introduce label propagation for the pseudolabeling. And [37] combines self-training with curriculum learning.…”

Spatial Consistency Loss for Training Multi-Label Classifiers from Single-Label Annotations

“…However, pseudolabels are prone to concept drift and confirmation bias, where early mislabeled samples lead to accumulating errors. Curriculum labeling [5] mitigates this using a refined training strategy. Noisy student [55] demonstrated state-ofthe-art results on ImageNet [27] using self-training and distillation on a large set of unlabeled images, by iterative relabeling data and using increasingly larger student models.…”

“…We show in Sec. 4.2 that simply considering expected positives as positives leads to unsatisfactory results, possibly due to label drift of those pseudo-labels, where early mislabeled samples lead to accumulating errors [5]. Our expected negative (EN) only applies a binary-cross entropy loss on annotated positives and the set of expected negatives, ignoring the expected positive labels in the loss.…”

Iterative Loop Method Combining Active and Semi-Supervised Learning for Domain Adaptive Semantic Segmentation

“…Semi-supervised learning (SSL) involves two typical paradigms: consistency regularization [3,48,73] and entropy minimization [5,6,27,49]. Consistency regularization forces the model to produce stable and consistent predictions on the same unlabeled data under various perturbations [71].…”