FlexPD: A Flexible Framework of First-Order Primal-Dual Algorithms for Distributed Optimization

Mansoori, Fatemeh; Wei, Ermin

doi:10.1109/tsp.2021.3083981

Cited by 11 publications

(11 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To achieve fast convergence, accelerated distributed gradient descent algorithms were presented in [38][39][40][41][42]. Meanwhile, the distributed primal-dual algorithms were also developed in [43]. Moreover, Newton's algorithms were developed in [44,45], and quasi-Newton methods were provided in [46].…”

Section: Dmentioning

confidence: 99%

A distributed gradient algorithm based on randomized block-coordinate and projection-free over networks

Zhu

Wang

Zhang

et al. 2022

Complex Intell. Syst.

View full text Add to dashboard Cite

The computational bottleneck in distributed optimization methods, which is based on projected gradient descent, is due to the computation of a full gradient vector and projection step. This is a particular problem for large datasets. To reduce the computational complexity of existing methods, we combine the randomized block-coordinate descent and the Frank–Wolfe techniques, and then propose a distributed randomized block-coordinate projection-free algorithm over networks, where each agent randomly chooses a subset of the coordinates of its gradient vector and the projection step is eschewed in favor of a much simpler linear optimization step. Moreover, the convergence performance of the proposed algorithm is also theoretically analyzed. Specifically, we rigorously prove that the proposed algorithm can converge to optimal point at rate of $${\mathcal {O}}(1/t)$$ O ( 1 / t ) under convexity and $${\mathcal {O}}(1/t^2)$$ O ( 1 / t 2 ) under strong convexity, respectively. Here, t is the number of iterations. Furthermore, the proposed algorithm can converge to a stationary point, where the “Frank-Wolfe” gap is equal to zero, at a rate $${\mathcal {O}}(1/\sqrt{t})$$ O ( 1 / t ) under non-convexity. To evaluate the computational benefit of the proposed algorithm, we use the proposed algorithm to solve the multiclass classification problems by simulation experiments on two datasets, i.e., aloi and news20. The results shows that the proposed algorithm is faster than the existing distributed optimization algorithms due to its lower computation per iteration. Furthermore, the results also show that well-connected graphs or smaller graphs leads to faster convergence rate, which can confirm the theoretical results.

show abstract

Section: Dmentioning

confidence: 99%

A distributed gradient algorithm based on randomized block-coordinate and projection-free over networks

Zhu

Wang

Zhang

et al. 2022

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…However, the minimization of a local objective function at each iteration is required for distributed ADMM methods, which leads to computational complexity and might break the balance between computation costs and performance. This bottleneck is overcome by References 25,26 through the use of a multi‐step communication strategy that performs multiple gradient descent steps and a dual step per iteration. By adjusting the number of gradient descent steps during each iteration, the algorithms in References 25,26 achieve a balance for computation or communication costs and performance tradeoffs.…”

Section: Introductionmentioning

confidence: 99%

“…This bottleneck is overcome by References 25,26 through the use of a multi‐step communication strategy that performs multiple gradient descent steps and a dual step per iteration. By adjusting the number of gradient descent steps during each iteration, the algorithms in References 25,26 achieve a balance for computation or communication costs and performance tradeoffs. It is worth noting that the multi‐step communication strategy proposed in the literature 25,26 is different from multi‐step distributed online learning 27 .…”

Section: Introductionmentioning

confidence: 99%

“…By adjusting the number of gradient descent steps during each iteration, the algorithms in References 25,26 achieve a balance for computation or communication costs and performance tradeoffs. It is worth noting that the multi‐step communication strategy proposed in the literature 25,26 is different from multi‐step distributed online learning 27 . Multi‐step distributed online learning uses the past multi‐step performance metrics as input for predicting the future multi‐step performance metrics online or offline, while multi‐step communication uses the past one‐step information as input to update the variable for the current step.…”

Section: Introductionmentioning

confidence: 99%

“…Next, we summarize the main contributions:A distributed primal‐dual algorithm with the stochastic averaging gradient is developed to reduce the cost of gradient evaluation. Unlike most existing algorithms 8‐12,14,21‐26 requiring costly evaluation of local batch gradients, for each node, the execution of the proposed algorithm only has to calculate the gradient of one randomly selected instantaneous function per iteration. A linear convergence rate of the proposed algorithm is established for strongly‐convex and continuously‐differentiable instantaneous functions and the upper bound of the fixed stepsizes is obtained. In addition, a novel analytical framework is proposed to analyze the convergence rate. …”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A stochastic averaging gradient algorithm with multi‐step communication for distributed optimization

Zheng

Feng

et al. 2023

Optim Control Appl Methods

View full text Add to dashboard Cite

This paper studies distributed convex optimization problems over an undirected network where all nodes cooperate to minimize a sum of local objective functions. Each local objective function is further assumed to be an average of several convex instantaneous functions. By incorporating the stochastic averaging gradient into the distributed first-order primal-dual method, a stochastic averaging gradient algorithm with multi-step communication is proposed to solve the optimization problem. For each node, one randomly selected gradient of an instantaneous function is evaluated per iteration, which effectively reduces the computation cost of the algorithm. Based on conditions of strong convexity and Lipschitz continuity of the local instantaneous functions, a linear convergence rate of the proposed algorithm is guaranteed. Numerical simulations on the logistic regression problem demonstrate the performance of the algorithm and correctness of the theoretical results.

show abstract

Beyond Distributed Training in the Cloud

Joshi

2022

Synthesis Lectures on Learning, Networks, and Algorithms

View full text Add to dashboard Cite

FlexPD: A Flexible Framework of First-Order Primal-Dual Algorithms for Distributed Optimization

Cited by 11 publications

References 41 publications

A distributed gradient algorithm based on randomized block-coordinate and projection-free over networks

A distributed gradient algorithm based on randomized block-coordinate and projection-free over networks

A stochastic averaging gradient algorithm with multi‐step communication for distributed optimization

Beyond Distributed Training in the Cloud

Contact Info

Product

Resources

About