Classification with imperfect training labels

Abstract

We study the effect of imperfect training data labels on the performance of classification methods. In a general setting, where the probability that an observation in the training dataset is mislabelled may depend on both the feature vector and the true label, we bound the excess risk of an arbitrary classifier trained with imperfect labels in terms of its excess risk for predicting a noisy label. This reveals conditions under which a classifier trained with imperfect labels remains consistent for classifying uncorrupted test data points. Furthermore, under stronger conditions, we derive detailed asymptotic properties for the popular k-nearest neighbour (knn), support vector machine (SVM) and linear discriminant analysis (LDA) classifiers. One consequence of these results is that the knn and SVM classifiers are robust to imperfect training labels, in the sense that the rate of convergence of the excess risks of these classifiers remains unchanged; in fact, our theoretical and empirical results even show that in some cases, imperfect labels may improve the performance of these methods. On the other hand, the LDA classifier is shown to be typically inconsistent in the presence of label noise unless the prior probabilities of each class are equal. Our theoretical results are supported by a simulation study.

Deep learning-based cross-classifications reveal conserved spatial behaviors within tumor histological images

“…For example, we train CNNs by associating all tiles within a slide with the same label, even though tumor slides will contain some regions that are non-tumor. Our approaches still work because classifiers can tolerate some error in the training data 52 . In the machine learning literature, this corresponds to the general problem of multi-label, multi-instance supervised learning with imbalanced data, an active area of research including for medical image data 28,[53][54][55] .…”

Random projections: Data perturbation for classification problems

“…There has been a fair amount of work recently on label noise (Frénay & Kabán, 2014; Frénay & Verleysen, 2014). Simple methods such as k ‐nearest neighbors and SVM have been shown to be robust to label noise (Cannings, Fan, & Samworth, 2019). It is less clear, however, how more sophisticated methods, such as those based on random projections, will be affected by noise.…”

Binary classification with corrupted labels

“…Manwani and Sastry [16] discuss the noisetolerance property of risk minimization. Cannings, Fan and Samworth [5] show that LDA is consistent under the noise, and Blanco, Japón and Puerto [2] propose robust algorithms that apply relabeling and clustering to SVM.…”

“…Finally, the general setting-where ρ(x, y) might vary with x-is studied by Cannings, Fan and Samworth [5]. In particular, they examine a setting for k-nearest neighbor where the corrupted labels Y i are more "clean" than the original labels Y i , in the sense that the corruption mechanism defined by ρ(x, y) acts to denoise labels near the decision boundary (i.e., η(x) ≈ 0.5) Specifically, suppose that, for values x with η(x) slightly higher than 0.5, we have ρ(x, +1) < ρ(x, −1) (that is, a label Y i = −1 that "should" instead be positive, has a greater chance of being flipped to Y i = +1), and similarly if η(x) is slightly lower than 0.5 then ρ(x, +1) > ρ(x, −1).…”