Progressive Stochastic Learning for Noisy Labels

Abstract

Large-scale learning problems require a plethora of labels that can be efficiently collected from crowdsourcing services at low cost. However, labels annotated by crowdsourced workers are often noisy, which inevitably degrades the performance of large-scale optimizations including the prevalent stochastic gradient descent (SGD). Specifically, these noisy labels adversely affect updates of the primal variable in conventional SGD. To solve this challenge, we propose a robust SGD mechanism called progressive stochastic learning (POSTAL), which naturally integrates the learning regime of curriculum learning (CL) with the update process of vanilla SGD. Our inspiration comes from the progressive learning process of CL, namely learning from "easy" tasks to "complex" tasks. Through the robust learning process of CL, POSTAL aims to yield robust updates of the primal variable on an ordered label sequence, namely, from "reliable" labels to "noisy" labels. To realize POSTAL mechanism, we design a cluster of "screening losses," which sorts all labels from the reliable region to the noisy region. To sum up, POSTAL using screening losses ensures robust updates of the primal variable on reliable labels first, then on noisy labels incrementally until convergence. In theory, we derive the convergence rate of POSTAL realized by screening losses. Meanwhile, we provide the robustness analysis of representative screening losses. Experimental results on UCI1 simulated and Amazon Mechanical Turk crowdsourcing data sets show that the POSTAL using screening losses is more effective and robust than several existing baselines.1UCI is the abbreviation of University of California Irvine.

Image classification with deep learning in the presence of noisy labels: A survey

“…Two types of loss corrections are proposed: forward and backward. Forward loss multiplies model predictions with noise transition matrix to match them with noisy labels while backward loss multiplies Noise Model Based Methods 1.Noisy Channel a.Extra layer: Linear fully connected layer [42][43], softmax layer [44] b.Separate Network: Estimating noise type [45], masking [46], quality embedding [47] c.Explicit Calculation: EM [26], [48], [49], conditional independence [50], forward&backward loss [51], unsupervised generative model [52], bayesian form [53] 2.Label Noise Cleansing a.Using Reference Set: train cleaner on reference set [54], [55], clean based on extracted features [56], teacher cleans for student [57], ensemble of networks [58] b.Not Using Reference Set: Moving average of network predictions [59], consistency loss [60], ensemble of network [61], prototypes [62], random split [63], confidence policy [64] 3.Sample Choosing a.Self Consistency: Consistency with model [65], consistency with moving average of model [66], graph-based [67], dividing to two subset [68] b.Curriculum Learning: Screening loss [69], teacher-student [70], selecting uncertain samples [71], extra layer with similarity matrix [72], curriculum loss [73], data complexity [74], partial labels [75] c.Multiple Classifiers: Consistency of networks [76], co-teaching [77]- [79] d.Active Learning: Relabel hard ...…”

Learning with Biased Complementary Labels

“…All results are shown in Table 1. Our loss modi cation ("LM") method is compared to a partial label (PL) method [6], a multi-label (ML) method [27], and "PC/S" (the pairwise-comparison formulation with sigmoid loss), which achieved the best performance in [12]. We can see, "PC/S" achieves very good performances.…”

An Efficient and Provable Approach for Mixture Proportion Estimation Using Linear Independence Assumption

“…Mixture proportion estimation has been an important ingredient for learning with label noise [31,19,36,5,39,13,12], learning with complementary labels [38], domain adaptation [11,41], semi-supervised learning [29], anomaly rejection [2,30], PU learning [8,23], and multiple instance learning [1], etc. Here, we give a brief summary of the former two applications, and show how the proposed method efficiently solves these problems.…”