Consistency Regularization on Clean Samples for Learning with Noisy Labels

Abstract

In the recent years, deep learning has achieved significant results in various areas of machine learning. Deep learning requires a huge amount of data to train a model, and data collection techniques such as web crawling have been developed. However, there is a risk that these data collection techniques may generate incorrect labels. If a deep learning model for image classification is trained on a dataset with noisy labels, the generalization performance significantly decreases. This problem is called Learning with Noisy Labels (LNL). One of the recent researches on LNL, called DivideMix [1], has successfully divided the dataset into samples with clean labels and ones with noisy labels by modeling loss distribution of all training samples with a two-component Mixture Gaussian model (GMM). Then it treats the divided dataset as labeled and unlabeled samples and trains the classification model in a semi-supervised manner. Since the selected samples have lower loss values and are easy to classify, training models are in a risk of overfitting to the simple pattern during training. To train the classification model without overfitting to the simple patterns, we propose to introduce consistency regularization on the selected samples by GMM. The consistency regularization perturbs input images and encourages model to outputs the same value to the perturbed images and the original images. The classification model simultaneously receives the samples selected as clean and their perturbed ones, and it achieves higher generalization performance with less overfitting to the selected samples. We evaluated our method with synthetically generated noisy labels on CIFAR-10 and CIFAR-100 and obtained results that are comparable or better than the state-of-the-art method.

Unbiased Pseudo-Labeling for Learning with Noisy Labels

“…DivideMix [7] is a pioneering method that combines sample selection and SSL for LNL and has achieved state-of-the-art performance in recent years by dividing the training data into a set of labeled and unlabeled samples and assigning pseudo-labels to the unlabeled samples, which is called co-divide. The development of advanced state-of-the-art methods based on DivideMix is an active research topic [8]. However, DivideMix, when dividing the training data using a model that has memorized noisy labels, can lead to bias in which the number of labels for certain classes either increases or decreases according to the errors in the memorized labels.…”