Federated Learning with Non-IID Data

Abstract

Federated learning enables resource-constrained edge compute devices, such as mobile phones and IoT devices, to learn a shared model for prediction, while keeping the training data local. This decentralized approach to train models provides privacy, security, regulatory and economic benefits. In this work, we focus on the statistical challenge of federated learning when local data is non-IID. We first show that the accuracy of federated learning reduces significantly, by up to ~55% for neural networks trained for highly skewed non-IID data, where each client device trains only on a single class of data. We further show that this accuracy reduction can be explained by the weight divergence, which can be quantified by the earth mover's distance (EMD) between the distribution over classes on each device and the population distribution. As a solution, we propose a strategy to improve training on non-IID data by creating a small subset of data which is globally shared between all the edge devices. Experiments show that accuracy can be increased by ~30% for the CIFAR-10 dataset with only 5% globally shared data.

A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learning

“…The first FL framework, Federated Averaging (Fe-dAvg) (McMahan et al 2017) glb for the next training round. Although FedAvg achieves superior performance on homogeneous clients with IID data, its performance degrades when data distribution among client devices is non-IID (Zhao et al 2018;Wang et al 2020a). While multiple FL frameworks were proposed to mitigate the impact of statistical diversity by either controlling the divergence between the local model and global model (Li et al 2020;Karimireddy et al 2019), or by training a personalized model for each client (Smith et al 2017;Deng, Kamani, and Mahdavi 2020;Zhang et al 2020), none of these studies has considered system performance objectives in their design.…”

FedGCN: Convergence and Communication Tradeoffs in Federated Training of Graph Convolutional Networks

“…However, FedAvg may not converge if data from different clients is non-i.i.d. (Zhao et al, 2018;Li et al, 2019) and some clients do not regularly participate in the training (Yang et al, 2021), as is often the case in federated learning scenarios. We show similar results for federated graph training.…”

“…This data, however, is often privacy-sensitive: for example, users in a social network may not want to reveal the websites they have visited. In non-graphical settings, distributed learning has recently shown promise for preserving user privacy while training accurate models, e.g., federated learning algorithms have become increasingly popular (Yang et al, 2021;Zhao et al, 2018). Some papers have begun to apply federated algo-1 Code available at https://github.com/yh-yao/Federated-GCN rithms to training GCNs (He et al, 2021;Wang et al, 2020).…”

“…The model, however, may still have difficulty converging, as federated learning may converge poorly when client data is non-i.i.d. (Zhao et al, 2018). It is thus unclear under which circumstances more communication is worth the convergence improvement.…”

Client Selection in Federated Learning under Imperfections in Environment

“…In ideal cases, Independent and Identically Distributed (IID) data are desired, both in terms of features as well as classes. However, in real scenarios, non-IID data varying from moderate to strongly skewed distribution is inevitable [5,6]. The global class imbalance problem may prevail and adversely affect the performance of the global model [7].…”