Asynchronous Distributed ADMM for Large-Scale Optimization—Part I: Algorithm and<?Pub _newline ?>Convergence Analysis

Chang, Tsung-Hui; Mei, Hong; Liao, Wei-Cheng; Wang, Xiangfeng

doi:10.1109/tsp.2016.2537271

Cited by 176 publications

(131 citation statements)

References 29 publications

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…Since all agents and parallel random walks keep individual clock, both PW-ADMM and IPW-ADMM are asynchronous algorithms. However, our proposed algorithms are different from existing work [15], [17], where only one master updates the variable z. Moreover, the updated z is only sent to the agents just active.…”

Section: B Intelligent Parallel Random Walk Admmmentioning

confidence: 94%

Decentralized Consensus Optimization Based on Parallel Random Walk

Chen

et al. 2020

IEEE Commun. Lett.

View full text Add to dashboard Cite

The alternating direction method of multipliers (ADMM) has recently been recognized as a promising approach for large-scale machine learning models. However, very few results study ADMM from the aspect of communication costs, especially jointly with running time. In this letter, we investigate the communication efficiency and running time of ADMM in solving the consensus optimization problem over decentralized networks. We first review the effort of random walk ADMM (W-ADMM), which reduces communication costs at the expense of running time. To accelerate the convergence speed of W-ADMM, we propose the parallel random walk ADMM (PW-ADMM) algorithm, where multiple random walks are active at the same time. Moreover, to further reduce the running time of PW-ADMM, the intelligent parallel random walk ADMM (IPW-ADMM) algorithm is proposed through integrating the Random Walk with Choice with PW-ADMM. By numerical results from simulations, we demonstrate that the proposed algorithms can be both communication efficient and fast in running speed compared with state-of-the-art methods.Index Terms-Decentralized network; consensus optimization; alternating direction method of multipliers (ADMM).

show abstract

Section: B Intelligent Parallel Random Walk Admmmentioning

confidence: 94%

Decentralized Consensus Optimization Based on Parallel Random Walk

Chen

et al. 2020

IEEE Commun. Lett.

View full text Add to dashboard Cite

show abstract

“…We consider the convergence of the proposed approach under the asynchronous protocol where the master has the freedom to make updates with gradients from only a partial set of worker machines. We start with introducing important conditions that are used commonly in previous work [13], [20], [40].…”

Section: Algorithmmentioning

confidence: 99%

Provably Communication-Efficient Asynchronous Distributed Inference for Convex and Nonconvex Problems

Ren

Haupt

2018

2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP)

View full text Add to dashboard Cite

This paper proposes and analyzes a communicationefficient distributed optimization framework for general nonconvex nonsmooth signal processing and machine learning problems under an asynchronous protocol. At each iteration, worker machines compute gradients of a known empirical loss function using their own local data, and a master machine solves a related minimization problem to update the current estimate. We prove that for nonconvex nonsmooth problems, the proposed algorithm converges with a sublinear rate over the number of communication rounds, coinciding with the best theoretical rate that can be achieved for this class of problems. Linear convergence is established without any statistical assumptions of the local data for problems characterized by composite loss functions whose smooth parts are strongly convex. Extensive numerical experiments verify that the performance of the proposed approach indeed improves -sometimes significantly -over other state-of-the-art algorithms in terms of total communication efficiency.

show abstract

“…Therefore, the speed of distributed ADMM is limited by the slowest worker nodes especially when the nodes have different computation and communication delays. This will create a new bottleneck that the conventional distributed ADMM suffers from low convergence speed and expensive time cost in practice [2], [23]. After that, local variables in the same group are used to generate group variable w Gj in (iii).…”

Section: B Distributed Admm Framework For Linear Classificationmentioning

confidence: 99%

“…Also, according to the different communication protocols, the distributed ADMM can be summarized into two classes: 1) asynchronous, where some nodes of the network are allowed to wake up at random and perform local updates in a noncoordinated fashion [23], [24] and 2) synchronous, where the master node is triggered only if it receives the information from all the slave nodes in each iteration [25]. In this paper, we only focus on the synchronous distributed ADMM under the master-slave mode.…”

mentioning

confidence: 99%

“…Although the previous distributed ADMM algorithms [25], [28] for linear classification have been validated for promising results, they still suffer from several limitations, from either theoretical or practical perspectives: 1) they are normally lacking in theoretical analysis, which is needed to deeply understand the scalability and generalization ability of the distributed algorithms and 2) they usually have low convergence speed and expensive communication cost [23], [33].…”

mentioning

confidence: 99%

See 1 more Smart Citation

Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification

Wang

Gao

Shi

et al. 2017

IEEE Trans. Cybern.

View full text Add to dashboard Cite

The alternating direction method of multipliers (ADMM) algorithm has been widely employed for distributed machine learning tasks. However, it suffers from several limitations, e.g., a relative low convergence speed, and an expensive time cost. To this end, in this paper, a novel method, namely the group-based ADMM (GADMM), is proposed for distributed linear classification. In particular, to accelerate the convergence speed and improve global consensus, a group layer is first utilized in GADMM to divide all the slave nodes into several groups. Then, all the local variables (from the slave nodes) are gathered in the group layer to generate different group variables. Finally, by using a weighted average method, the group variables are coordinated to update the global variable (from the master node) until the solution of the global problem is reached. According to the theoretical analysis, we found that: 1) GADMM can mathematically converge at the rate , where is the number of outer iterations and 2) by using the grouping methods, GADMM can improve the convergence speed compared with the distributed ADMM framework without grouping methods. Moreover, we systematically evaluate GADMM on four publicly available LIBSVM datasets. Compared with disADMM and stochastic dual coordinate ascent with alternating direction method of multipliers-ADMM, for distributed classification, GADMM is able to reduce the number of outer iterations, which leads to faster convergence speed and better global consensus. In particular, the statistical significance test has been experimentally conducted and the results validate that GADMM can significantly save up to 30% of the total time cost (with less than 0.6% accuracy loss) compared with disADMM on large-scale datasets, e.g., webspam and epsilon.

show abstract

Asynchronous Distributed ADMM for Large-Scale Optimization—Part I: Algorithm andConvergence Analysis

Cited by 176 publications

References 29 publications

Decentralized Consensus Optimization Based on Parallel Random Walk

Decentralized Consensus Optimization Based on Parallel Random Walk

Provably Communication-Efficient Asynchronous Distributed Inference for Convex and Nonconvex Problems

Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification

Contact Info

Product

Resources

About