Accelerating incremental gradient optimization with curvature information

Wai, Hoi To; Shi, Wei; Uribe, César A.; Nedić, Angelia; Scaglione, Anna

doi:10.1007/s10589-020-00183-1

Cited by 10 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because of asynchrony, the right-hand side of the inequalities will involve delayed versions of V k and W k that perturb the convergence of the synchronous iteration. For example, in the analysis of the incremental aggregated gradient method [27], accelerated incremental aggregated gradient method with curvature information [67], asynchronous quasi-Newton method [21], and asynchronous forward-backward method for solving monotone inclusion problems [60], one can establish iterate relationships on the form…”

Section: Novel Sequence Results For Asynchronous Iterationsmentioning

confidence: 99%

Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees

Feyzmahdavian¹,

Johansson²

2021

Preprint

View full text Add to dashboard Cite

We introduce novel convergence results for asynchronous iterations which appear in the analysis of parallel and distributed optimization algorithms. The results are simple to apply and give explicit estimates for how the degree of asynchrony impacts the convergence rates of the iterates. Our results shorten, streamline and strengthen existing convergence proofs for several asynchronous optimization methods, and allow us to establish convergence guarantees for popular algorithms that were thus far lacking a complete theoretical understanding. Specifically, we use our results to derive better iteration complexity bounds for proximal incremental aggregated gradient methods, to provide less conservative analyses of the speedup conditions for asynchronous block-coordinate implementations of Krasnosel'ski ȋ-Mann iterations, and to quantify the convergence rates for totally asynchronous iterations under various assumptions on communication delays and update rates.

show abstract

Section: Novel Sequence Results For Asynchronous Iterationsmentioning

confidence: 99%

Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees

Feyzmahdavian¹,

Johansson²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Extensions of PIAG that allow for a non-smooth regularizer include [2,3,12] for convex f and [13,14] for non-convex f . In addition, a recent work [15] compensates for the information delays in PIAG using Hessian information. However, all these papers use an upper bound of the worst-case delay to determine the step-size.…”

Section: Algorithms and Related Workmentioning

confidence: 99%

“…In addition, because τ k ∈ [0, k] ∀k ∈ N 0 , τ 0 = 0 and γ 0 = αγ ′ . Substituting these into (47) yields (15).…”

Section: Proof Of Theoremmentioning

confidence: 99%

Delay-adaptive step-sizes for asynchronous learning

Wu¹,

Magnússon²,

Feyzmahdavian³

et al. 2022

Preprint

View full text Add to dashboard Cite

In scalable machine learning systems, model training is often parallelized over multiple nodes that run without tight synchronization. Most analysis results for the related asynchronous algorithms use an upper bound on the information delays in the system to determine learning rates. Not only are such bounds hard to obtain in advance, but they also result in unnecessarily slow convergence. In this paper, we show that it is possible to use learning rates that depend on the actual time-varying delays in the system. We develop general convergence results for delay-adaptive asynchronous iterations and specialize these to proximal incremental gradient descent and block-coordinate descent algorithms. For each of these methods, we demonstrate how delays can be measured on-line, present delay-adaptive step-size policies, and illustrate their theoretical and practical advantages over the state-of-the-art.

show abstract

Proximal variable smoothing method for three-composite nonconvex nonsmooth minimization with a linear operator

Liu,

Xia

2023

Numer Algor

View full text Add to dashboard Cite

Accelerating incremental gradient optimization with curvature information

Cited by 10 publications

References 19 publications

Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees

Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees

Delay-adaptive step-sizes for asynchronous learning

Proximal variable smoothing method for three-composite nonconvex nonsmooth minimization with a linear operator

Contact Info

Product

Resources

About