Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization

Nedić, Angelia; Olshevsky, Alex; Rabbat, Michael

doi:10.1109/jproc.2018.2817461

Cited by 428 publications

(339 citation statements)

References 111 publications

Supporting

Mentioning

329

Contrasting

Order By: Relevance

“…Comparing (43), (42) and (44) with the KKT conditions (14), (15) and (16), we conclude that the triple (x k , z k , φ k ) satisfies the KKT conditions when k goes to infinity. Next, we show that {(x k , z k , φ k )} converges when k → ∞.…”

Section: Appendix B Proof Of Theoremmentioning

confidence: 84%

Communication-Censored Linearized ADMM for Decentralized Consensus Optimization

Yaohua

Tian

et al. 2020

IEEE Trans. on Signal and Inf. Process. over Networks

View full text Add to dashboard Cite

In this paper, we propose a communication-and computation-efficient algorithm to solve a convex consensus optimization problem defined over a decentralized network. A remarkable existing algorithm to solve this problem is the alternating direction method of multipliers (ADMM), in which at every iteration every node updates its local variable through combining neighboring variables and solving an optimization subproblem. The proposed algorithm, called as communicationcensored linearized ADMM (COLA), leverages a linearization technique to reduce the iteration-wise computation cost of ADMM and uses a communication-censoring strategy to alleviate the communication cost. To be specific, COLA introduces successive linearization approximations to the local cost functions such that the resultant computation is first-order and light-weight. Since the linearization technique slows down the convergence speed, COLA further adopts the communicationcensoring strategy to avoid transmissions of less informative messages. A node is allowed to transmit only if the distance between the current local variable and its previously transmitted one is larger than a censoring threshold. COLA is proven to be convergent when the local cost functions have Lipschitz continuous gradients and the censoring threshold is summable. When the local cost functions are further strongly convex, we establish the linear (sublinear) convergence rate of COLA, given that the censoring threshold linearly (sublinearly) decays to 0. Numerical experiments corroborate with the theoretical findings and demonstrate the satisfactory communication-computation tradeoff of COLA. . 2 that approximates the second-order information with local gradients. The lower complexity bounds and rate-optimal algorithms of decentralized optimization are developed in [36]- [38]. Note that the communication cost in the aforementioned algorithms is proportional to the number of iterations, since after a given number of iterations every node needs to communicate with its neighbors.In all decentralized algorithms, there is an essential communication-computation tradeoff [39]- [43]. An algorithm with light-weight iteration-wise computation generally needs more number of iterations, and in consequence more communication cost, to reach a target accuracy. For example, compared with ADMM, DLM enjoys simple gradient-based computation, but suffers from relatively slow convergence speed and high communication cost. In this paper, we aim at achieving a favorable communication-computation tradeoff in a decentralized network, where the nodes are only affordable to light-weight gradient-based computation. The limitation on the computation power may come from that the nodes are equipped with cheap computing units in a wireless sensor network, or from that using higher-order information is prohibitively time-consuming for finding a high-dimensional solution in a machine learning system.Given the constraint on the computation cost, we adopt the communication-censoring strategy to further save the commu...

show abstract

Section: Appendix B Proof Of Theoremmentioning

confidence: 84%

Communication-Censored Linearized ADMM for Decentralized Consensus Optimization

Yaohua

Tian

et al. 2020

IEEE Trans. on Signal and Inf. Process. over Networks

View full text Add to dashboard Cite

show abstract

“…In this section we review the distributed subgradient method that has been proposed in the pioneering works [11,12] (see also the tutorial papers [2,3,4]). In this survey, we report a proof based on the analysis proposed in the references above.…”

Section: Distributed Subgradient Methodsmentioning

confidence: 99%

“…It also discusses tailored, parallel and distributed optimization algorithms based on decomposition techniques and including also the distributed subgradient method. Recent surveys analyze thoroughly average consensus [2] and the distributed subgradient method [2,3,4], with a literature review on other distributed optimization techniques. The book [5] provides parallel and distributed asynchronous optimization algorithms, including gradient tracking techniques.…”

Section: Scope Of the Monographmentioning

confidence: 99%

“…We point out that one may also consider strongly connected directed graphs that admits a doubly-stochastic weighted adjacency matrix. Detailed convergence analyses of distributed subgradient schemes have been provided, e.g., in [11,12,2,4,3]. For the sake of completeness, this survey provides a proof for the convergence of Algorithm 1 that is mainly inspired by the references above.…”

Section: Algorithm 1 Distributed Subgradientmentioning

confidence: 99%

See 1 more Smart Citation

Distributed Optimization for Smart Cyber-Physical Networks

Notarstefano¹,

Notarnicola²,

Camisa³

2019

View full text Add to dashboard Cite

The presence of embedded electronics and communication capabilities as well as sensing and control in smart devices has given rise to the novel concept of cyber-physical networks, in which agents aim at cooperatively solving complex tasks by local computation and communication. Numerous estimation, learning, decision and control tasks in smart networks involve the solution of large-scale, structured optimization problems in which network agents have only a partial knowledge of the whole problem. Distributed optimization aims at designing local computation and communication rules for the network processors allowing them to cooperatively solve the global optimization problem without relying on any central unit. The purpose of this survey is to provide an introduction to distributed optimization methodologies. Principal approaches, namely (primal) consensus-based, duality-based and constraint exchange methods, are formalized. An analysis of the basic schemes is supplied, and state-of-the-art extensions are reviewed.

show abstract

“…One basic research problem is the leaderless consensus problem, where the agents achieve a common value of interest by interacting with their local neighbors. The consensus algorithm initiates the research trend in the area and triggers a lot of applications including formation [4], distributed optimization [5], synchronization of biochemical networks [6], and cooperative adaptive identification [7].…”

Section: Introductionmentioning

confidence: 99%

Distributed Consensus for Multiple Lagrangian Systems With Parametric Uncertainties and External Disturbances Under Directed Graphs

Mei

2020

IEEE Trans. Control Netw. Syst.

View full text Add to dashboard Cite

In this paper, we study the leaderless consensus problem for multiple Lagrangian systems in the presence of parametric uncertainties and external disturbances under directed graphs. For achieving asymptotic behavior, a robust continuous term with adaptive varying gains is added to alleviate the effects of the external disturbances with unknown bounds. In the case of a fixed directed graph, by introducing an integrate term in the auxiliary variable design, the final consensus equilibrium can be explicitly derived. We show that the agents achieve weighted average consensus, where the final equilibrium is dependent on three factors, namely, the interactive topology, the initial positions of the agents, and the control gains of the proposed control algorithm. In the case of switching directed graphs, a model reference adaptive consensus based algorithm is proposed such that the agents achieve leaderless consensus if the infinite sequence of switching graphs is uniformly jointly connected. Motivated by the fact that the relative velocity information is difficult to obtain accurately, we further propose a leaderless consensus algorithm with gain adaptation for multiple Lagrangian systems without using neighbors' velocity information. We also propose a model reference adaptive consensus based algorithm without using neighbors' velocity information for switching directed graphs. The proposed algorithms are distributed in the sense of using local information from its neighbors and using no comment control gains. Numerical simulations are performed to show the effectiveness of the proposed algorithms. where he is currently an Associate Professor. His current research interests include coordination of distributed multi-agent systems and its applications on formation of unmanned vehicles and robots.

show abstract

Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization

Cited by 428 publications

References 111 publications

Communication-Censored Linearized ADMM for Decentralized Consensus Optimization

Communication-Censored Linearized ADMM for Decentralized Consensus Optimization

Distributed Optimization for Smart Cyber-Physical Networks

Distributed Consensus for Multiple Lagrangian Systems With Parametric Uncertainties and External Disturbances Under Directed Graphs

Contact Info

Product

Resources

About