Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning

Abstract

Federated learning (FL) is a distributed machine learning architecture that leverages a large number of workers to jointly learn a model with decentralized data. FL has received increasing attention in recent years thanks to its data privacy protection, communication efficiency and a linear speedup for convergence in training (i.e., convergence performance increases linearly with respect to the number of workers). However, existing studies on linear speedup for convergence are only limited to the assumptions of i.i.d. datasets across workers and/or full worker participation, both of which rarely hold in practice. So far, it remains an open question whether or not the linear speedup for convergence is achievable under non-i.i.d. datasets with partial worker participation in FL. In this paper, we show that the answer is affirmative. Specifically, we show that the federated averaging (FedAvg) algorithm (with two-sided learning rates) on non-i.i.d. datasets in non-convex settings achieves a convergence rate O( 1 √ mKT + 1 T ) for full worker participation and a convergence rate O( 1 √ nKT + 1 T ) for partial worker participation, where K is the number of local steps, T is the number of total communication rounds, m is the total worker number and n is the worker number in one communication round if for partial worker participation. Our results also reveal that the local steps in FL could help the convergence and show that the maximum number of local steps can be improved to T /m. We conduct extensive experiments on MNIST and CIFAR-10 to verify our theoretical results.

FedGCN: Convergence and Communication Tradeoffs in Federated Training of Graph Convolutional Networks

“…However, FedAvg may not converge if data from different clients is non-i.i.d. (Zhao et al, 2018;Li et al, 2019) and some clients do not regularly participate in the training (Yang et al, 2021), as is often the case in federated learning scenarios. We show similar results for federated graph training.…”

“…Assumptions 5.1 and 5.2 are standard in the federated learning literature (Yang et al, 2021). We then introduce another assumption to quantify the information loss: Assumption 5.3.…”

“…This data, however, is often privacy-sensitive: for example, users in a social network may not want to reveal the websites they have visited. In non-graphical settings, distributed learning has recently shown promise for preserving user privacy while training accurate models, e.g., federated learning algorithms have become increasingly popular (Yang et al, 2021;Zhao et al, 2018). Some papers have begun to apply federated algo-1 Code available at https://github.com/yh-yao/Federated-GCN rithms to training GCNs (He et al, 2021;Wang et al, 2020).…”

Communication-Efficient Device Scheduling for Federated Learning Using Stochastic Optimization

“…Both [19] and [20] consider convergence, but only for strongly convex loss functions. Convergence results for non-convex loss functions with partial device participation have also been presented in [7], [21], [22], but they only consider the case where devices are chosen uniformly at random with or without replacement and do not allow for arbitrary selection probabilities. Finally, [23] considers arbitrary probabilities for each device, but these probabilities are held constant throughout training and are not reflected in the parameter aggregation weights.…”

Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling

“…This is because if (17) holds with an inequality, we can always find an R < R * that satisfies (17) with equality, but the solution (q * , R ) can further reduce the objective function value. Therefore, for the optimal R, (17) always holds, and we can obtain R from (17) and substitute it into the objective of Problem P2. Then, the objective of Problem P2 is…”