FedCorr: Multi-Stage Federated Learning for Label Noise Correction

Abstract

Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables clients to jointly train a global model. In real-world FL implementations, client data could have label noise, and different clients could have vastly different label noise levels. Although there exist methods in centralized learning for tackling label noise, such methods do not perform well on heterogeneous label noise in FL settings, due to the typically smaller sizes of client datasets and data privacy requirements in FL. In this paper, we propose FedCorr, a general multi-stage framework to tackle heterogeneous label noise in FL, without making any assumptions on the noise models of local clients, while still maintaining client data privacy. In particular, (1) FedCorr dynamically identifies noisy clients by exploiting the dimensionalities of the model prediction subspaces independently measured on all clients, and then identifies incorrect labels on noisy clients based on persample losses. To deal with data heterogeneity and to increase training stability, we propose an adaptive local proximal regularization term that is based on estimated local noise levels. (2) We further finetune the global model on identified clean clients and correct the noisy labels for the remaining noisy clients after finetuning. (3) Finally, we apply the usual training on all clients to make full use of all local data. Experiments conducted on CIFAR-10/100 with federated synthetic label noise, and on a real-world noisy dataset, Clothing1M, demonstrate that FedCorr is robust to label noise and substantially outperforms the state-ofthe-art methods at multiple noise levels.

Federated Learning Design and FunctionalModels: Survey

“…In this method, the client selection is performed based on the random selection criteria. Additionally, the Power-of-Choice [39] method has been used for managing biased client selection and has received better convergence rates compared to the random client selection method. Better performance in terms of faster convergence and more consistent management is made possible using methods like arbitrary client selection [44] and adaptive client selection [45] by assessing and updating probabilities pertaining to the client relationships.…”

“…With the help of this framework, the local data efficiency is seen to improve, and the communication cost is reduced, while overall improving end-to-end performance. Another framework, the Fedcorr [39] multi-stage selection framework, used for managing noise of local clients while independently managing all the clients, has been noted for increasing the fairness of the federated learning process by adjusting the client selection strategies and overall improving the efficiency of the results. The DRFL [40] method is used to improve the weight accorded to each client and thereby increase the fairness of the result.…”

FedCoop: Cooperative Federated Learning for Noisy Labels

“…Although DNNs are prone to overfitting with noisy label samples, the memorization effect can still be leveraged to facilitate learning from clean samples in a simple pattern during the initial stage of training, before gradually memorizing the noisy samples [1]. Based on this effect, various studies [34,13,18,33] have filtered noisy samples based on their loss values. However, these methods overlook the overfitting property of DNNs, which severely decreases the discriminatory capability of empirical loss [1,36].…”

“…However, they are not directly transferable to FL for the two fundamental issues: 1) highly heterogeneous noise distribution among clients causes local model divergence that cannot be addressed by robust loss functions [22]; 2) sample selection approaches for centralized setting are infeasible when dealing with limited training data on each client, which overfit the noisy data and lead to unstable performance. Recently, several methods have been proposed to mitigate the problem of noisy labels in FL [26,6,35,34,33]. These methods mainly focus on obtaining data with clean labels by modeling noise probability or utilizing the memorization effect of Deep Neural Networks (DNNs).…”

“…Federated Learning with Noisy Label. Existing works about FL with noisy labels can be summarized into two main categories: 1) Benchmark data-dependent methods [9,26,6] and 2) Label correction-based methods [33,34,35]. The Benchmark datadependent methods extract the subset of clean clients or data with clean labels using public benchmark data.…”