Communication-Efficient Network-Distributed Optimization with Differential-Coded Compressors

Zhang, Xin; Liu, Jia; Zhu, Zhengyuan; Bentley, Elizabeth Serena

doi:10.1109/infocom41043.2020.9155432

Cited by 2 publications

(1 citation statement)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar compression ideas have also been extended to decentralized learning over networks without dedicated parameter servers. In [39,40], Zhang et al developed a series of decentralized learning algorithms with differential-coded compressions. Koloskova et al [14] proposed an algorithm that could achieve a linear speedup with respect to the number of workers for convergence in decentralized learning with arbitrary gradient compression.…”

Section: Introductionmentioning

confidence: 99%

CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning

Yang¹,

Liu²,

Bentley³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Federated learning (FL) is a prevailing distributed learning paradigm, where a large number of workers jointly learn a model without sharing their training data. However, high communication costs could arise in FL due to large-scale (deep) learning models and bandwidth-constrained connections. In this paper, we introduce a communication-efficient algorithmic framework called CFedAvg for FL with non-i.i.d. datasets, which works with general (biased or unbiased) SNR-constrained compressors. We analyze the convergence rate of CFedAvg for non-convex functions with constant and decaying learning rates. The CFedAvg algorithm can achieve an O (1/ √ + 1/ ) convergence rate with a constant learning rate, implying a linear speedup for convergence as the number of workers increases, where is the number of local steps, is the number of total communication rounds, and is the total worker number. This matches the convergence rate of distributed/federated learning without compression, thus achieving high communication efficiency while not sacrificing learning accuracy in FL. Furthermore, we extend CFedAvg to cases with heterogeneous local steps, which allows different workers to perform a different number of local steps to better adapt to their own circumstances. The interesting observation in general is that the noise/variance introduced by compressors does not affect the overall convergence rate order for non-i.i.d. FL. We verify the effectiveness of our CFedAvg algorithm on three datasets with two gradient compression schemes of different compression ratios. CCS CONCEPTS• Computing methodologies → Distributed algorithms; • Theory of computation → Distributed algorithms.

show abstract

Section: Introductionmentioning

confidence: 99%