Multi-class Probabilistic Bounds for Self-learning

Abstract

Self-learning is a classical approach for learning with both labeled and unlabeled observations which consists in giving pseudo-labels to unlabeled training instances with a confidence score over a predetermined threshold. At the same time, the pseudo-labeling technique is prone to error and runs the risk of adding noisy labels into unlabeled training data. In this paper, we present a probabilistic framework for analyzing self-learning in the multi-class classification scenario with partially labeled data. First, we derive a transductive bound over the risk of the multi-class majority vote classifier. Based on this result, we propose to automatically choose the threshold for pseudo-labeling that minimizes the transductive bound. Then, we introduce a mislabeling error model to analyze the error of the majority vote classifier in the case of the pseudo-labeled data. We derive a probabilistic C-bound over the majority vote error when an imperfect label is given. Empirical results on different data sets show the effectiveness of our framework compared to several state-of-the-art semi-supervised approaches.1 It is also known as self-training or self-labeling.

Self-Training: A Survey

“…A theoretical perspective for future work is to study the self-training method by taking explicitly into account the fact that the final classifier is learned on the training data with noisy labels. Some attempts have been already initiated to model this label noise using the Massart noise [Hadjadj et al, 2021] or the mislabeling transition matrix [Feofanov et al, 2021], but there is a clear need for more research in this direction.…”