Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

Abstract

Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human annotation constraints. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudolabels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100 and Mini-ImageNet despite being much simpler than other state-of-the-art. These results demonstrate that pseudo-labeling can outperform consistency regularization methods, while the opposite was supposed in previous work. Source code is available at https://git.io/fjQsC.

LaplaceNet: A Hybrid Energy-Neural Model for Deep Semi-Supervised Classification

“…Additional, dependent on the accuracy of the pseudo-labels, we increase the amount of labelled data the model has access to and reduce overfitting to the initally small label set. There are many ways to incorporate unlabelled data / pseudo-label pairs into the loss function but the most common ways are to either create a specific loss term for the unlabelled data pseudo-label pairs [12], [18] or by using composite batches containing both labelled and unlabelled data and keeping the standard supervised classification loss [20], [33].…”

“…As pointed out by Arazo et al [20] there is a potential pitfall in this style of approach. Networks are often wrong and the neural network can overfit to its own incorrectly guessed pseudo-labels in a process termed confirmation bias.…”

“…1) Methods which used the 13-CNN architecture [32]: Π-Model [30], Mean Teacher(MT) [32], Virtual Adversarial Training (VAT) [31], Label Propogation for Deep Semi-Supervised Learning (LP) [33], Smooth Neighbors on Teacher Graphs (SNTG) [42], Stochastic Weight Averaging(SWA) [43], Interpolation Consistency Training (ICT) [21], Dual Student [44], Transductive Semi-Supervised Deep Learning(TSSDL) [35], Density-Aware Graphs (DAG) [38] and Pseudo-Label Mixup [20]. Unfortunately, due to the natural progress in the field, each paper has different implementation choices which are not standardised.…”

“…Recent approaches in SSL have proposed costly optimisation schemes involving multi-term loss functions to improve the generalisation of their models [14], [20]. Some approaches [12] use separate loss terms for unlabelled and labelled data, whilst consistency regularisation approaches such as [13] use a standard supervised loss in combination with a specialised consistency loss.…”

“…Some approaches [12] use separate loss terms for unlabelled and labelled data, whilst consistency regularisation approaches such as [13] use a standard supervised loss in combination with a specialised consistency loss. Other approaches go even further [14], [20] and use three or more loss terms which promote entropy minimisation, class balancing or simultaneously minimise several consistency losses.…”