Improving the Transient Times for Distributed Stochastic Gradient Methods

Huang, Kun; Pu, Shi

doi:10.48550/arxiv.2105.04851

Cited by 5 publications

(9 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Related Worksupporting

confidence: 84%

“…Simultaneously and independently, a recent work [41] has established a similar result to this work. However, [41] studies the transient stage of D 2 /Exact-Diffusion for the strongly-convex scenario only, but not for the generally-convex scenario. We also prove a lower bound of D-SGD with homogeneous dataset that shows that D 2 /Exact-Diffusion's dependence on network topology cannot be worse than D-SGD and always better under the heterogeneous setting.…”

Section: Related Worksupporting

confidence: 84%

See 1 more Smart Citation

Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD

Yuan

Alghunaim

2021

Preprint

View full text Add to dashboard Cite

We consider decentralized stochastic optimization problems where a network of agents each owns a local cost function cooperate to find a minimizer of the global-averaged cost. A widely studied decentralized algorithm for this problem is D-SGD in which each node applies a stochastic gradient descent step, then averages its estimate with its neighbors. D-SGD is attractive due to its efficient single-iteration communication and can achieve linear speedup in convergence (in terms of the network size). However, D-SGD is very sensitive to the network topology. For smooth objective functions, the transient stage (which measures how fast the algorithm can reach the linear speedup stage) of D-SGD is on the order of) for strongly convex and generally convex cost functions, respectively, where 1 − β ∈ (0, 1) is a topology-dependent quantity that approaches 0 for a large and sparse network.Hence, D-SGD suffers from slow convergence for large and sparse networks.In this work, we study the non-asymptotic convergence property of the D 2 /Exact-diffusion algorithm.By eliminating the influence of data heterogeneity between nodes, D 2 /Exact-diffusion is shown to have an enhanced transient stage that are on the order of O(n/(1 − β)) and O(n 3 /(1 − β) 2 ) for strongly convex and generally convex cost functions, respectively. Moreover, we provide a lower bound of the transient stage of D-SGD under homogeneous data distributions, which coincides with the transient stage of D 2 /Exact-diffusion in the strongly-convex setting. These results show that removing the influence of data heterogeneity can ameliorate the network topology dependence of D-SGD. Compared with existing decentralized algorithms bounds, D 2 /Exact-diffusion is least sensitive to network topology .

show abstract

Section: Related Worksupporting

confidence: 84%

Section: Related Worksupporting

confidence: 84%

Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD

Yuan

Alghunaim

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…where the second line follows from (15). In light of the convexity of h and Jensen's inequality, for all t ≥ 1 we have that h(z t+1 ) ≤ 1 n n i=1 h(z i t+1 ) and hence (53) implies…”

Section: B Proof Of Lemma 5 B1 Step 1: Descent Inequality For the Con...mentioning

confidence: 95%

“…For this smooth formulation, variants of decentralized stochastic gradient descent (DSGD), e.g., [4,26,52,70], admit simple implementations yet provide competitive practical performance against centralized methods in homogeneous environments like data centers. When the data distributions across the network become heterogeneous, the performance of DSGD in both practice and theory degrades significantly [15,39,57,59,68]. To address this issue, stochastic methods that are robust to heterogeneous data have been proposed, e.g., D2 [51] that is derived from primal-dual formulations [22,25,47,69] and GT-DSGD [29,63] that is based on gradient tracking [10,33,38,41,67].…”

Section: Literature Reviewmentioning

confidence: 99%

A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency

Xin,

Das,

Khan

et al. 2021

Preprint

View full text Add to dashboard Cite

Decentralized optimization is a promising parallel computation paradigm for large-scale data analytics and machine learning problems defined over a network of nodes. This paper is concerned with decentralized non-convex composite problems with population or empirical risk. In particular, the networked nodes are tasked to find an approximate stationary point of the average of local, smooth, possibly non-convex risk functions plus a possibly non-differentiable extended valued convex regularizer. Under this general formulation, we propose the first provably efficient, stochastic proximal gradient framework, called ProxGT. Specifically, we construct and analyze several instances of ProxGT that are tailored respectively for different problem classes of interest. Remarkably, we show that the sample complexities of these instances are network topology-independent and achieve linear speedups compared to that of the corresponding centralized optimal methods implemented on a single node. Contents

show abstract

“…One line of research proposes new algorithms that are less sensitive to topologies. For example, [66,23,65,57,1] removed data heterogeneity with bias-correction techniques in [68,29,62,40,69], and [14,61,7,27] utilized periodic global averaging or multiple partial averaging steps. All these methods have improved topology dependence.…”

Section: Related Workmentioning

confidence: 99%

Exponential Graph is Provably Efficient for Decentralized Deep Training

Ying¹,

Yuan²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel SGD to inexact averaging. The less exact the averaging is, however, the more the total iterations the training needs to take. Therefore, the key to making decentralized SGD efficient is to realize nearly-exact averaging using little communication. This requires a skillful choice of communication topology, which is an under-studied topic in decentralized optimization. In this paper, we study so-called exponential graphs where every node is connected to O(log(n)) neighbors and n is the total number of nodes. This work proves such graphs can lead to both fast communication and effective averaging simultaneously. We also discover that a sequence of log(n) one-peer exponential graphs, in which each node communicates to one single neighbor per iteration, can together achieve exact averaging. This favorable property enables one-peer exponential graph to average as effective as its static counterpart but communicates more efficiently. We apply these exponential graphs in decentralized (momentum) SGD to obtain the state-of-the-art balance between per-iteration communication and iteration complexity among all commonly-used topologies. Experimental results on a variety of tasks and models demonstrate that decentralized (momentum) SGD over exponential graphs promises both fast and highquality training. Our code is implemented through BlueFog and available at https://github.com/Bluefog-Lib/NeurIPS2021-Exponential-Graph.

show abstract

Improving the Transient Times for Distributed Stochastic Gradient Methods

Cited by 5 publications

References 37 publications

Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD

Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD

A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency

Exponential Graph is Provably Efficient for Decentralized Deep Training

Contact Info

Product

Resources

About