DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training

Yuan, Kun; Chen, Yiming; Huang, Xinmeng; Pan, Pan; Xu, Yinghui; Yin, Wotao

doi:10.1109/iccv48922.2021.00302

Cited by 18 publications

(25 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A limitation of the D 2 algorithm is that it is not clear how it can be applied to time-varying topologies, and that it can only be used on constant mixing topologies with negative eigenvalue bounded from below by − 1 3 . Other authors proposed algorithms that perform well on heterogeneous DL tasks [25,59], but theoretical proofs that these algorithms are independent of the degree of heterogeneity are still pending.…”

Section: Pcεmentioning

confidence: 99%

An Improved Analysis of Gradient Tracking for Decentralized Machine Learning

Koloskova¹,

Lin²,

Stich³

2022

Preprint

View full text Add to dashboard Cite

We consider decentralized machine learning over a network where the training data is distributed across n agents, each of which can compute stochastic model updates on their local data. The agent's common goal is to find a model that minimizes the average of all local loss functions. While gradient tracking (GT) algorithms can overcome a key challenge, namely accounting for differences between workers' local data distributions, the known convergence rates for GT algorithms are not optimal with respect to their dependence on the mixing parameter p (related to the spectral gap of the connectivity matrix). We provide a tighter analysis of the GT method in the stochastic strongly convex, convex and non-convex settings. We improve the dependency on p from O(p −2 ) to O(p −1 c −1 ) in the noiseless case and from O(p −3/2 ) to O(p −1/2 c −1 ) in the general stochastic case, where c ≥ p is related to the negative eigenvalues of the connectivity matrix (and is a constant in most practical applications). This improvement was possible due to a new proof technique which could be of independent interest.

show abstract

Section: Pcεmentioning

confidence: 99%

An Improved Analysis of Gradient Tracking for Decentralized Machine Learning

Koloskova¹,

Lin²,

Stich³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Communication efficiency. When the network topology is sparse (e.g., a ring or a one-peer exponential graph [3], [33]), each partial averaging step (5) incurs O(1) latency and O(1) transmission time (the inverse of bandwidth), which are independent of n. Since each node only synchronizes with its direct neighbors, there is low synchronization overhead. effective in aggregating information than global averaging, some decentralized algorithms can match or exceed the performance of global-averaging-based distributed algorithms: [1], [29] established that decentralized SGD can achieve the same asymptotic linear speedup in convergence rate as (parameter server based) distributed SGD; [3], [33] used exponential graph topologies to realize both efficient communication and effective aggregation by partial averaging; [37], [38], [31], [39] improved the convergence rate of decentralized SGD by removing data heterogeneity between nodes; [40], [4], [30], [41] enhanced the effectiveness of partial averaging by periodically calling global averaging.…”

Section: A Concepts and Theoretical Foundationsmentioning

confidence: 99%

“…They can, however, obtain the same results through local dynamics, namely, a series of computation and agent-to-agent direct communication steps. On large-scale optimization tasks involving distributed datasets, recent decentralized computational methods have shown strong, sometimes superior, performance [1], [2], [3], [4], [5].…”

Section: Introductionmentioning

confidence: 99%

“…Comparing the performances of these two types of algorithms is not merely a task of comparing their costs-per-iteration (though we do in this paper) but also the numbers of iterations or total times for reaching a solution of the same accuracy. The theoretical analyses of iteration complexities and total running times are out of the scope of this paper; see [1], [3], [29], [30], [4], [5], [31] for some recent decentralized algorithms that are proven faster than distributed algorithms. This paper, however, provides some numerical comparison results that demonstrate BlueFog.…”

Section: Introductionmentioning

confidence: 99%

“…It keeps evolving during the past two years, and the progress was reported in some conferences 1 . BlueFog has provided demos for several distributed and decentralized algorithms introduced in [32], and it has supported all deep learning experiments in [4], [5], [33]. All BlueFog code and supporting documents can be found at https://github.com/Bluefog-Lib.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning

Ying¹,

Yuan²,

Hu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Decentralized algorithm is a form of computation that achieves a global goal through local dynamics that relies on low-cost communication between directly-connected agents. On large-scale optimization tasks involving distributed datasets, decentralized algorithms have shown strong, sometimes superior, performance over distributed algorithms with a central node. Recently, developing decentralized algorithms for deep learning has attracted great attention. They are considered as low-communication-overhead alternatives to those using a parameter server or the Ring-Allreduce protocol. However, the lack of an easy-to-use and efficient software package has kept most decentralized algorithms merely on paper. To fill the gap, we introduce BlueFog, a python library for straightforward, high-performance implementations of diverse decentralized algorithms. Based on a unified abstraction of various communication operations, BlueFog offers intuitive interfaces to implement a spectrum of decentralized algorithms, from those using a static, undirected graph for synchronous operations to those using dynamic and directed graphs for asynchronous operations. BlueFog also adopts several system-level acceleration techniques to further optimize the performance on the deep learning tasks. On mainstream DNN training tasks, BlueFog reaches a much higher throughput and achieves an overall 1.2× ∼ 1.8× speedup over Horovod, a state-of-the-art distributed deep learning package based on Ring-Allreduce. BlueFog is open source at https://github.com/Bluefog-Lib/bluefog.

show abstract

A unified momentum-based paradigm of decentralized SGD for non-convex models and heterogeneous data

Du,

Cheng,

2024

Artificial Intelligence

View full text Add to dashboard Cite

DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training

Cited by 18 publications

References 22 publications

An Improved Analysis of Gradient Tracking for Decentralized Machine Learning

An Improved Analysis of Gradient Tracking for Decentralized Machine Learning

BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning

A unified momentum-based paradigm of decentralized SGD for non-convex models and heterogeneous data

Contact Info

Product

Resources

About