Centrality and Consistency: Two-Stage Clean Samples Identification for Learning with Instance-Dependent Noisy Labels

Abstract

Deep models trained with noisy labels are prone to overfitting and struggle in generalization. Most existing solutions are based on an ideal assumption that the label noise is class-conditional, i.e. instances of the same class share the same noise model, and are independent of features. While in practice, the real-world noise patterns are usually more fine-grained as instance-dependent ones, which poses a big challenge, especially in the presence of inter-class imbalance. In this paper, we propose a two-stage clean samples identification method to address the aforementioned challenge. First, we employ a class-level feature clustering procedure for the early identification of clean samples that are near the class-wise prediction centers. Notably, we address the class imbalance problem by aggregating rare classes according to their prediction entropy. Second, for the remaining clean samples that are close to the ground truth class boundary (usually mixed with the samples with instance-dependent noises), we propose a novel consistency-based classification method that identifies them using the consistency of two classifier heads: the higher the consistency, the larger the probability that a sample is clean. Extensive experiments on several challenging benchmarks demonstrate the superior performance of our method against the stateof-the-art. Code is available at https://github.com/uitrbn/TSCSI_IDN.

Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance

“…In the context of learning from noisy labels, the key concept is to leverage these dynamics as criteria for identifying and separating noisy samples. Several works propose to identify samples with lower training loss as the clean subset Han et al, 2018;Jiang et al, 2018;Zhao et al, 2022;Wang et al, 2023b), however, they are generally simplistic and inflexible, resulting in the selection of only easy samples. To address this limitation, alternative approaches have been proposed to effectively utilize the loss or confidences during training, as demonstrated in (Zhang et al, 2021a) and (Nishi et al, 2021).…”