Auto-weighted Robust Federated Learning with Corrupted Data Sources

Abstract

Federated learning provides a communication-efficient and privacy-preserving training process by enabling learning statistical models with massive participants without accessing their local data. Standard federated learning techniques that naively minimize an average loss function are vulnerable to data corruptions from outliers, systematic mislabeling, or even adversaries. In this paper, we address this challenge by proposing Auto-weighted Robust Federated Learning ( ARFL ), a novel approach that jointly learns the global model and the weights of local updates to provide robustness against corrupted data sources. We prove a learning bound on the expected loss with respect to the predictor and the weights of clients, which guides the definition of the objective for robust federated learning. We present an objective that minimizes the weighted sum of empirical risk of clients with a regularization term, where the weights can be allocated by comparing the empirical risk of each client with the average empirical risk of the best p clients. This method can downweight the clients with significantly higher losses, thereby lowering their contributions to the global model. We show that this approach achieves robustness when the data of corrupted clients is distributed differently from the benign ones. To optimize the objective function, we propose a communication-efficient algorithm based on the blockwise minimization paradigm. We conduct extensive experiments on multiple benchmark datasets, including CIFAR-10, FEMNIST, and Shakespeare, considering different neural network models. The results show that our solution is robust against different scenarios including label shuffling, label flipping, and noisy features, and outperforms the state-of-the-art methods in most scenarios.

An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning

“…For the IID partitioning, we randomly split the training set into 20 subsets and allocate them to the 20 clients, thus each client has 2500 training samples. For Non-IID partition, we follow prior work [34], [35] and model the Non-IID data distributions with a Dirichlet distribution p l ∼ Dir K (α), then we allocate a p l,k proportion of the training sample of class l to client k, in which a smaller α indicates stronger Non-IID data partition. We let α = 0.1 and visualize the resulting statistical heterogeneity of labels in Fig.…”

An Experimental Study of Byzantine-Robust Aggregation Schemes in Federated Learning

“…For the IID partitioning, we randomly split the training set into 20 subsets and allocate them to the 20 clients. For the Non-IID partition, we follow prior work [38], [39] and model the Non-IID data distributions with a Dirichlet distribution p l ∼ Dir K (α). Then we allocate a p l,k proportion of the training sample of class l to client k, in which a smaller α indicates a stronger Non-IID data partition.…”

“…Similar to other schemes, ClippedClustering is also less robust in Non-IID data scenarios. We note that Non-IIDness is a well-known challenge to FL especially when it comes to robustness [39], as it becomes hard to induce a consensus model for the benign clients if their data distributions are significantly different. Thus the malicious clients can damage the global model more easily.…”

GAN-Driven Data Poisoning Attacks and Their Mitigation in Federated Learning Systems

“…To achieve this, a GAN is trained over leaf blight images only, with a subset of nodes that correspond only to the malicious clients, namely 30% of the whole FL system. We selected this number because a significant number of comparable studies that assess defense mechanisms or suggest attacks employ a percentage similar to this one for the malevolent clients [58][59][60]. The GAN has been trained for 80 rounds using the joint malicious dataset.…”