Convergence Rate of Distributed ADMM Over Networks

Makhdoumi, Ali; Ozdaglar, Asuman

doi:10.1109/tac.2017.2677879

Cited by 167 publications

(147 citation statements)

References 69 publications

Supporting

Mentioning

146

Contrasting

Order By: Relevance

“…Proof Since our C‐ADMM is a direct application of ADMM to the dual problem (), we can similarly obtain that as t → ∞ , λ i ( t )→ λ ∗ , λ i ( t )→ λ j ( t ), for any

i \in V, j \in scriptN false(i false)

. Therefore, we still remain to show that any limit point of ( x 1 ( t +1),…, x N ( t +1)) is asymptotically optimal to (), ie, as t → ∞ ,

\nabla C_{i} ((), x_{i} false(t + 1 false)) + λ^{*} \to 0,

true \sum_{i = 1}^{N} x_{i} false(t + 1 false) - true \sum_{i = 1}^{N} B_{i} \to 0 .

To show , consider the optimality condition of (15a), ie,

\begin{matrix} 0 & = \nabla C_{i} false(x_{i} false(t + 1 false) false) + \frac{1}{M_{i} β} ((), x_{i} false(t + 1 false) - B_{i} - \sum_{j \in scriptN false(i false)} p_{j} false(t false) A_{j i} + β \sum_{j \in scriptN false(i false)} A_{j i}^{2} λ_{i} false(t false) - β \sum_{j \in scriptN false(i false)} A_{j i} y_{j} false(t false)) \\ = \nabla C_{i} false(x_{i} false(t + 1 false) false) + λ_{i} fal... \end{matrix}

…”

Section: Distributed Dc‐admmmentioning

confidence: 89%

“…We will use an implementation of ADMM algorithm, which separates each constraint associated with a node into multiple constraints that involve only the variable associated with one of the neighboring nodes. We use a reformulation technique introduced in the work of Bertsekas and Tsitsiklis to separate optimization variables in a constraint, allowing them to be updated simultaneously.…”

Section: Network Model and Problem Formulationmentioning

confidence: 99%

“…Following a similar argument as in deriving ADMM algorithm, we obtain the following update steps at each iteration: for any 1 ≤ j ≤ N , we have

λ_{j} false(t + 1 false) = {normalargmin}_{λ_{j}} ([], {normalΦ}_{j} false(λ_{j} false) + λ_{j} B_{j} + \sum_{i \in scriptN false(j false)} μ_{i j} false(t false) A_{i j} λ_{j} + \frac{β}{2} {((), A_{i j} λ_{j} - z_{i j} false(t false))}^{2});

for any 1 ≤ i ≤ N and

j \in scriptN false(i false)

, where p i ( t +1) is the Lagrange multiplier chosen to satisfy the constraint

\sum_{j \in scriptN false(i false)} z_{i j} false(t + 1 false) = 0

. By using the optimal solution z ij ( t +1), we have

z_{i j} false(t + 1 false) = A_{i j} λ_{j} false(t + 1 false) + \frac{μ_{i j} false(t false) - p_{i} false(t + 1 false)}{β} .

Taking the summation of for all

j \in scriptN false(i false)

, we obtain

\sum_{j \in N (i)} z_{i j} (t + 1) = 0 = \sum_{j \in N}

…”

Section: Network Model and Problem Formulationmentioning

confidence: 99%

“…Similar to distributed ADMM, the C‐ADMM algorithm is summarized as follows:

λ_{i} false(t + 1 false) = {normalargmin}_{λ_{i}} ({}, {normalΦ}_{i} false(λ_{i} false) + λ_{i} B_{i} + \sum_{j \in scriptN false(i false)} p_{j} false(t false) A_{j i} λ_{i} + \frac{β}{2} \sum_{j \in scriptN false(i false)} {([], y_{j} false(t false) + A_{j i} ((), λ_{i} - λ_{i} false(t false)))}^{2}),

y_{i} false(t + 1 false) = \frac{1}{d_{i} + 1} \sum_{j \in scriptN false(i false)} A_{j i} λ_{j} false(t + 1 false),

p_{i} false(t + 1 false) = p_{i} false(t false) + β y_{i} false(t + 1 false) .

…”

Section: Network Model and Problem Formulationmentioning

confidence: 99%

“…In particular, ADMM was found to converge linearly on a large class of problems, meaning a certain optimality measure can decrease by a constant fraction in each iteration of the algorithm. Makhdoumi and Ozdaglar extended the distributed ADMM that allows for a general choice of communication weight matrix, which is used to combine the iteration at different nodes and show that when functions are convex, converge with rate

O false(\frac{1}{T} false)

. Chang et al proposed two algorithms named the inexact consensus ADMM and the inexact dual consensus ADMM (IDC‐ADMM), where an inexact method reduces the computational complexity of consensus ADMM.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Distributed inexact dual consensus ADMM for network resource allocation

Long

Wang

et al. 2019

Optim Control Appl Methods

View full text Add to dashboard Cite

Summary This paper investigates two novel distributed algorithms based on alternating direction method of multipliers (ADMM) for network resource allocation of N agents. The main objective is to derive an optimal allocation that minimizes a global objective expressed as a sum of locally known separable convex objective functions. Based on a communication matrix, the dual resource allocation problem is changed into a consensus optimization problem, in which each agent broadcasts the outcome of its local processing to all his neighbors. In this paper, we first propose a new distributed dual consensus ADMM (DC‐ADMM) algorithm to address this consensus problem. Moreover, by applying an inexact step for each ADMM update, a distributed inexact DC‐ADMM (IDC‐ADMM) is also developed, which enables agents to perform cheap computation at each iteration. Finally, numerical simulations are delivered to illustrate and validate the proposed algorithm.

show abstract

“…Proof Since our C‐ADMM is a direct application of ADMM to the dual problem (), we can similarly obtain that as t → ∞ , λ i ( t )→ λ ∗ , λ i ( t )→ λ j ( t ), for any

i \in V, j \in scriptN false(i false)

. Therefore, we still remain to show that any limit point of ( x 1 ( t +1),…, x N ( t +1)) is asymptotically optimal to (), ie, as t → ∞ ,

\nabla C_{i} ((), x_{i} false(t + 1 false)) + λ^{*} \to 0,

true \sum_{i = 1}^{N} x_{i} false(t + 1 false) - true \sum_{i = 1}^{N} B_{i} \to 0 .

To show , consider the optimality condition of (15a), ie,

\begin{matrix} 0 & = \nabla C_{i} false(x_{i} false(t + 1 false) false) + \frac{1}{M_{i} β} ((), x_{i} false(t + 1 false) - B_{i} - \sum_{j \in scriptN false(i false)} p_{j} false(t false) A_{j i} + β \sum_{j \in scriptN false(i false)} A_{j i}^{2} λ_{i} false(t false) - β \sum_{j \in scriptN false(i false)} A_{j i} y_{j} false(t false)) \\ = \nabla C_{i} false(x_{i} false(t + 1 false) false) + λ_{i} fal... \end{matrix}

…”

Section: Distributed Dc‐admmmentioning

confidence: 89%

Section: Network Model and Problem Formulationmentioning

confidence: 99%

“…Following a similar argument as in deriving ADMM algorithm, we obtain the following update steps at each iteration: for any 1 ≤ j ≤ N , we have

λ_{j} false(t + 1 false) = {normalargmin}_{λ_{j}} ([], {normalΦ}_{j} false(λ_{j} false) + λ_{j} B_{j} + \sum_{i \in scriptN false(j false)} μ_{i j} false(t false) A_{i j} λ_{j} + \frac{β}{2} {((), A_{i j} λ_{j} - z_{i j} false(t false))}^{2});

for any 1 ≤ i ≤ N and

j \in scriptN false(i false)

, where p i ( t +1) is the Lagrange multiplier chosen to satisfy the constraint

\sum_{j \in scriptN false(i false)} z_{i j} false(t + 1 false) = 0

. By using the optimal solution z ij ( t +1), we have

z_{i j} false(t + 1 false) = A_{i j} λ_{j} false(t + 1 false) + \frac{μ_{i j} false(t false) - p_{i} false(t + 1 false)}{β} .

Taking the summation of for all

j \in scriptN false(i false)

, we obtain

\sum_{j \in N (i)} z_{i j} (t + 1) = 0 = \sum_{j \in N}

…”

Section: Network Model and Problem Formulationmentioning

confidence: 99%

“…Similar to distributed ADMM, the C‐ADMM algorithm is summarized as follows:

λ_{i} false(t + 1 false) = {normalargmin}_{λ_{i}} ({}, {normalΦ}_{i} false(λ_{i} false) + λ_{i} B_{i} + \sum_{j \in scriptN false(i false)} p_{j} false(t false) A_{j i} λ_{i} + \frac{β}{2} \sum_{j \in scriptN false(i false)} {([], y_{j} false(t false) + A_{j i} ((), λ_{i} - λ_{i} false(t false)))}^{2}),

y_{i} false(t + 1 false) = \frac{1}{d_{i} + 1} \sum_{j \in scriptN false(i false)} A_{j i} λ_{j} false(t + 1 false),

p_{i} false(t + 1 false) = p_{i} false(t false) + β y_{i} false(t + 1 false) .

…”

Section: Network Model and Problem Formulationmentioning

confidence: 99%

O false(\frac{1}{T} false)

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Distributed inexact dual consensus ADMM for network resource allocation

Long

Wang

et al. 2019

Optim Control Appl Methods

View full text Add to dashboard Cite

show abstract

Communication-efficient algorithms for decentralized and stochastic optimization

2018

View full text Add to dashboard Cite

We present a new class of decentralized first-order methods for nonsmooth and stochastic optimization problems defined over multiagent networks. Considering that communication is a major bottleneck in decentralized optimization, our main goal in this paper is to develop algorithmic frameworks which can significantly reduce the number of inter-node communications. We first propose a decentralized primal-dual method which can find an ǫ-solution both in terms of functional optimality gap and feasibility residual in O(1/ǫ) inter-node communication rounds when the objective functions are convex and the local primal subproblems are solved exactly. Our major contribution is to present a new class of decentralized primal-dual type algorithms, namely the decentralized communication sliding (DCS) methods, which can skip the inter-node communications while agents solve the primal subproblems iteratively through linearizations of their local objective functions. By employing DCS, agents can still find an ǫ-solution in O(1/ǫ) (resp., O(1/ √ ǫ)) communication rounds for general convex functions (resp., strongly convex functions), while maintaining the O(1/ǫ 2 ) (resp., O(1/ǫ)) bound on the total number of intra-node subgradient evaluations. We also present a stochastic counterpart for these algorithms, denoted by SDCS, for solving stochastic optimization problems whose objective function cannot be evaluated exactly. In comparison with existing results for decentralized nonsmooth and stochastic optimization, we can reduce the total number of inter-node communication rounds by orders of magnitude while still maintaining the optimal complexity bounds on intra-node stochastic subgradient evaluations. The bounds on the (stochastic) subgradient evaluations are actually comparable to those required for centralized nonsmooth and stochastic optimization under certain conditions on the target accuracy.Address(es) of author(s) should be given √ ǫ)) communication rounds while maintaining the O(1/ǫ 2 ) (resp., O(1/ǫ)) bound on the total number of intra-node subgradient evaluations when the objective functions are general convex (resp., strongly convex). The bounds on the subgradient evaluations are actually comparable to those optimal complexity bounds required for centralized nonsmooth optimization under certain conditions on the target accuracy, and hence are not improvable in general.Thirdly, we present a stochastic decentralized communication sliding method, denoted by SDCS, for solving stochastic optimization problems and show complexity bounds similar to those of DCS on the total number of required communication rounds and stochastic subgradient evaluations. In particular, only O(1/ǫ) (resp., O(1/ √ ǫ))

show abstract

A two-level distributed algorithm for nonconvex constrained optimization

Sun

2022

Comput Optim Appl

View full text Add to dashboard Cite

This paper aims to develop distributed algorithms for nonconvex optimization problems with complicated constraints associated with a network. The network can be a physical one, such as an electric power network, where the constraints are nonlinear power flow equations, or an abstract one that represents constraint couplings between decision variables of different agents. Despite the recent development of distributed algorithms for nonconvex programs, highly complicated constraints still pose a significant challenge in theory and practice. We first identify some difficulties with the existing algorithms based on the alternating direction method of multipliers (ADMM) for dealing with such problems. We then propose a reformulation that enables us to design a two-level algorithm, which embeds a specially structured three-block ADMM at the inner level in an augmented Lagrangian method framework. Furthermore, we prove the global and local convergence as well as iteration complexity of this new scheme for general nonconvex constrained programs, and show that our analysis can be extended to handle more complicated multi-block inner-level problems. Finally, we demonstrate with computation that the new scheme provides convergent and parallelizable algorithms for various nonconvex applications, and is able to complement the performance of the state-of-the-art distributed algorithms in practice by achieving either faster convergence in optimality gap or in feasibility or both.

show abstract

Convergence Rate of Distributed ADMM Over Networks

Cited by 167 publications

References 69 publications

Distributed inexact dual consensus ADMM for network resource allocation

Distributed inexact dual consensus ADMM for network resource allocation

Communication-efficient algorithms for decentralized and stochastic optimization

A two-level distributed algorithm for nonconvex constrained optimization

Contact Info

Product

Resources

About