On the O(1=k) convergence of asynchronous distributed alternating Direction Method of Multipliers

Wei, Ermin; Ozdaglar, Asuman

doi:10.1109/globalsip.2013.6736937

Cited by 300 publications

(303 citation statements)

References 43 publications

Supporting

Mentioning

299

Contrasting

Order By: Relevance

“…(Compare the last equation with (35).) The remainder of the proof proceeds analogously to that of Theorem 5.…”

Section: Proofs Of Theorems 7 Andmentioning

confidence: 97%

“…In addition to distributed gradient-like methods, a different type of methods -distributed (augmented) Lagrangian and distributed alternating direction of multipliers methods (ADMM) have been studied, e.g., in [10], [29]- [35]. They have in general more complex iterations than gradient methods, but may have a lower total communication cost, e.g., [30].…”

Section: Brief Comment On the Literaturementioning

confidence: 99%

“…Get: (34) where . Dividing (34) by and unwinding the resulting inequality, get: (35) Next, using Assumption 5, obtain, :…”

Section: Proof Of Theoremmentioning

confidence: 99%

“…They have in general more complex iterations than gradient methods, but may have a lower total communication cost, e.g., [30]. [34] shows convergence of an asynchronous ADMM algorithm while [35] shows an rate (in expectation) of an asynchronous ADMM method studied therein.…”

Section: Brief Comment On the Literaturementioning

confidence: 99%

See 3 more Smart Citations

Convergence Rates of Distributed Nesterov-Like Gradient Methods on Random Networks

Jakovetić

Xavier²,

Moura

2014

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

Abstract-We consider distributed optimization in random networks where nodes cooperatively minimize the sum of their individual convex costs. Existing literature proposes distributed gradient-like methods that are computationally cheap and resilient to link failures, but have slow convergence rates. In this paper, we propose accelerated distributed gradient methods that 1) are resilient to link failures; 2) computationally cheap; and 3) improve convergence rates over other gradient methods. We model the network by a sequence of independent, identically distributed random matrices drawn from the set of symmetric, stochastic matrices with positive diagonals. The network is connected on average and the cost functions are convex, differentiable, with Lipschitz continuous and bounded gradients. We design two distributed Nesterov-like gradient methods that modify the D-NG and D-NC methods that we proposed for static networks. We prove their convergence rates in terms of the expected optimality gap at the cost function. Let and be the number of per-node gradient evaluations and per-node communications, respectively. Then the modified D-NG achieves rates and , and the modified D-NC rates and , where is arbitrarily small. For comparison, the standard distributed gradient method cannot do better than and , on the same class of cost functions (even for static networks). Simulation examples illustrate our analytical findings.

show abstract

“…(Compare the last equation with (35).) The remainder of the proof proceeds analogously to that of Theorem 5.…”

Section: Proofs Of Theorems 7 Andmentioning

confidence: 97%

Section: Brief Comment On the Literaturementioning

confidence: 99%

“…Get: (34) where . Dividing (34) by and unwinding the resulting inequality, get: (35) Next, using Assumption 5, obtain, :…”

Section: Proof Of Theoremmentioning

confidence: 99%

Section: Brief Comment On the Literaturementioning

confidence: 99%

See 2 more Smart Citations

Convergence Rates of Distributed Nesterov-Like Gradient Methods on Random Networks

Jakovetić

Xavier²,

Moura

2014

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

show abstract

“…Based on the analysis in [25], one can easily extend the algorithm described above to its asynchronous counterpart with similar convergence guarantees.…”

Section: Remark 7 (Asynchronous Variant)mentioning

confidence: 99%

Distributed Online Modified Greedy Algorithm for Networked Storage Operation Under Uncertainty

Qin

Chow

Yang

et al. 2015

IEEE Trans. Smart Grid

View full text Add to dashboard Cite

Abstract-The integration of intermittent and stochastic renewable energy resources requires increased flexibility in the operation of the electric grid. Storage, broadly speaking, provides the flexibility of shifting energy over time; network, on the other hand, provides the flexibility of shifting energy over geographical locations. The optimal control of storage networks in stochastic environments is an important open problem. The key challenge is that, even in small networks, the corresponding constrained stochastic control problems on continuous spaces suffer from curses of dimensionality, and are intractable in general settings. For large networks, no efficient algorithm is known to give optimal or provably near-optimal performance for this problem. This paper provides an efficient algorithm to solve this problem with performance guarantees. We study the operation of storage networks, i.e., a storage system interconnected via a power network. An online algorithm, termed Online Modified Greedy algorithm, is developed for the corresponding constrained stochastic control problem. A sub-optimality bound for the algorithm is derived, and a semidefinite program is constructed to minimize the bound. In many cases, the bound approaches zero so that the algorithm is near-optimal. A task-based distributed implementation of the online algorithm relying only on local information and neighbor communication is then developed based on the alternating direction method of multipliers. Numerical examples verify the established theoretical performance bounds, and demonstrate the scalability of the algorithm.

show abstract

Asynchronous Parallel Computing

Yan¹

2021

Wiley StatsRef: Statistics Reference Online

View full text Add to dashboard Cite

Many single‐core statistical algorithms do not scale well to learn the large‐scale data that we encounter. With the emergence of multiple core central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), and supercomputers, parallel algorithms are developed to take advantage of multiple cores. However, parallel computing brings new challenges and bottlenecks. One major bottleneck is the synchronization among all the cores. Synchronization requires the cores that finish early to wait for other cores, and the waiting time could be very long when the number of cores is large. Asynchronous parallel computing lets the cores continue without waiting for others. However, the naive way of merely removing the synchronization step does not work in many cases. Adapting one algorithm to the asynchronous parallel environment is not trivial and usually requires careful reformulation. Moreover, the convergence of these asynchronous algorithms is challenging to show. In this article, we introduce several ways to implement asynchronous statistical algorithms. More specifically, we introduce asynchronous parallel coordinate algorithms that update the coordinates in parallel and asynchronous stochastic algorithms that evaluate the data samples in parallel.

show abstract

On the O(1=k) convergence of asynchronous distributed alternating Direction Method of Multipliers

Cited by 300 publications

References 43 publications

Convergence Rates of Distributed Nesterov-Like Gradient Methods on Random Networks

Convergence Rates of Distributed Nesterov-Like Gradient Methods on Random Networks

Distributed Online Modified Greedy Algorithm for Networked Storage Operation Under Uncertainty

Asynchronous Parallel Computing

Contact Info

Product

Resources

About