CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning

Yang, Haibo; Liu, Jia; Bentley, Elizabeth Serena

doi:10.23919/wiopt52861.2021.9589061

Cited by 29 publications

(34 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Consequently, we do not observe communication savings by performing multiple local updates at the clients, except in the special case when σ G = 0 (see Table 1). Similar observations have been made for minimization Yang et al (2021); Jhunjhunwala et al (2022) and very recently for minimax problems Yang et al (2022a). • In the absence of multiple local updates (i.e., τ i = 1 for all i) and with full participation (P = n), the resulting error O…”

Section: Nonsupporting

confidence: 76%

“…Since its introduction, FL has been an active area of research, with some remarkable successes Li et al (2020); . Research has shown practical benefits of, and provided theoretical justifications for commonly used practical techniques, such as, multiple local updates at the clients Stich (2018); Khaled et al (2020); Koloskova et al (2020); , partial client participation Yang et al (2021), communication compression Hamer et al (2020; Chen et al (2021). Further, impact of heterogeneity in the clients' local data Zhao et al (2018); Sattler et al (2019), as well as their system capabilities ; Mitra et al (2021) has been studied.…”

Section: Introductionmentioning

confidence: 99%

“…Compared to , we consider a more challenging problem and achieve higher communication savings (Table 1). Yang et al (2021) analyzed partial client participation in FL and demonstrated the theoretical benefit of using separate client/server learning rates. Deng & Mahdavi (2021); Sharma et al (2022) studied minimax problems in the federated setting but assumed a homogeneous number of local updates, with full client participation.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Federated Minimax Optimization with Client Heterogeneity

Sharma¹,

Panda²,

Joshi³

2023

Preprint

View full text Add to dashboard Cite

Minimax optimization has seen a surge in interest with the advent of modern applications such as GANs, and it is inherently more challenging than simple minimization. The difficulty is exacerbated by the training data residing at multiple edge devices or clients, especially when these clients can have heterogeneous datasets and local computation capabilities. We propose a general federated minimax optimization framework that subsumes such settings and several existing methods like Local SGDA. We show that naive aggregation of heterogeneous local progress results in optimizing a mismatched objective function -a phenomenon previously observed in standard federated minimization.To fix this problem, we propose normalizing the client updates by the number of local steps undertaken between successive communication rounds. We analyze the convergence of the proposed algorithm for classes of nonconvexconcave and nonconvex-nonconcave functions and characterize the impact of heterogeneous client data, partial client participation, and heterogeneous local computations. Our analysis works under more general assumptions on the intra-client noise and inter-client heterogeneity than so far considered in the literature. For all the function classes considered, we significantly improve the existing computation and communication complexity results. Experimental results support our theoretical claims.

show abstract

Section: Nonsupporting

confidence: 76%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Federated Minimax Optimization with Client Heterogeneity

Sharma¹,

Panda²,

Joshi³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Much effort has been made in optimizing FL and covers a variety of perspectives including communication [23], [24], update rules [25], [26], [27], [28], flexible aggregation [4], [29] and personalization [30], [31]. The control of device participation is imperative in crossdevice FL scenarios [32], [33] where the quality of local data is uncontrollable and the clients show varied value for the training task [6]. However, little attention has been paid to the problems caused by low-quality data and their impact on FL's efficiency and effectiveness.…”

Section: Related Workmentioning

confidence: 99%

FedProf: Selective Federated Learning based on Distributional Representation Profiling

Lin

et al. 2023

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Federated Learning (FL) has shown great potential as a privacy-preserving solution to learning from decentralized data that are only accessible to end devices (i.e., clients). The data locality constraint offers strong privacy protection but also makes FL sensitive to the condition of local data. Apart from statistical heterogeneity, a large proportion of the clients, in many scenarios, are probably in possession of low-quality data that are biased, noisy or even irrelevant. As a result, they could significantly slow down the convergence of the global model we aim to build and also compromise its quality. In light of this, we first present a new view of local data by looking into the representation space and observing that they converge in distribution to Normal distributions before activation. We provide theoretical analysis to support our finding. Further, we propose FEDPROF, a novel algorithm for optimizing FL over non-IID data of mixed quality. The key of our approach is a distributional representation profiling and matching scheme that uses the global model to dynamically profile data representations and allows for low-cost, lightweight representation matching. Using the scheme we sample clients adaptively in FL to mitigate the impact of low-quality data on the training process. We evaluated our solution with extensive experiments on different tasks and data conditions under various FL settings. The results demonstrate that the selective behavior of our algorithm leads to a significant reduction in the number of communication rounds and the amount of time (up to 2.4× speedup) for the global model to converge and also provides accuracy gain.

show abstract

“…Federated learning (FL) is a powerful distributed training paradigm for modern large-scale machine learning [1,2,10,11,13,16,22,33,[35][36][37]. FL leverages a large number of workers to collaboratively learn a global model.…”

Section: Introductionmentioning

confidence: 99%

NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Data

Zhang¹,

Fang²,

Liu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) has received a surge of interest in recent years thanks to its benefits in data privacy protection, efficient communication, and parallel data processing. Also, with appropriate algorithmic designs, one could achieve the desirable linear speedup for convergence effect in FL. However, most existing works on FL are limited to systems with i.i.d. data and centralized parameter servers and results on decentralized FL with heterogeneous datasets remains limited. Moreover, whether or not the linear speedup for convergence is achievable under fully decentralized FL with data heterogeneity remains an open question. In this paper, we address these challenges by proposing a new algorithm, called NET-FLEET, for fully decentralized FL systems with data heterogeneity. The key idea of our algorithm is to enhance the local update scheme in FL (originally intended for communication efficiency) by incorporating a recursive gradient correction technique to handle heterogeneous datasets. We show that, under appropriate parameter settings, the proposed NET-FLEET algorithm achieves a linear speedup for convergence. We further conduct extensive numerical experiments to evaluate the performance of the proposed NET-FLEET algorithm and verify our theoretical findings. CCS CONCEPTS• Computing methodologies → Machine learning.

show abstract

CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning

Cited by 29 publications

References 4 publications

Federated Minimax Optimization with Client Heterogeneity

Federated Minimax Optimization with Client Heterogeneity

FedProf: Selective Federated Learning based on Distributional Representation Profiling

NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Data

Contact Info

Product

Resources

About