Global Convergence Rate of Proximal Incremental Aggregated Gradient Methods

Vanli, N. Denizcan; Gürbüzbalaban, Mert; Ozdaglar, Asuman

doi:10.1137/16m1094415

Cited by 36 publications

(28 citation statements)

References 31 publications

(34 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, Lemma 3.2 from [16] shows that the sequence {h (k) } k≥1 converges if 1− √ µγ+O(γ For large m, the rate in (4.34) is worse than the rate of 1 − O(1/(mκ(F ))) analyzed in [29] for IAG. The analysis above shows that the theoretical convergence rate may not be improved using the acceleration technique that we have applied in A-CIAG, as long as the same analysis framework for A-CIAG was adopted.…”

Section: Discussionmentioning

confidence: 97%

“…As analyzed in [16,12], due to the dependence of m 2 on the right hand side, the sequence of squared norm { θ k − θ ⋆ 2 } k≥1 converges only when γ = O(1/m), and finally this shows that θ k − θ ⋆ 2 converges linearly at rate 1 − O(1/(m 2 κ(F ))). Notice that a recent work [29] has strengthened this rate to 1 − O(1/(mκ(F ))) and it is possible to further improve the rate through studying the expected convergence when the component function selection is random, e.g., [27].…”

Section: Gradient Tracking In Incrementalmentioning

confidence: 99%

“…To this end, a popular yet powerful approach is to adopt the so-called incremental (or stochastic) methods where only one of the component functions, e.g., the i k th one, f i k (θ), is explored at the kth iteration. Examples include the incremental gradient (IG) [6], incremental aggregated gradient (IAG) [7,29,16] methods when i k is deterministic; and the stochastic gradient (SG) [25], stochastic average gradient (SAG) [27], SAGA [11], stochastic variance reduced gradient (SVRG) [32] methods when i k is chosen randomly. Furthermore, related work can be found in [19,28] for the case with non-convex functions; in [17] for the case of primal-dual optimization; in [21,22] for the case with non-smooth functions.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Curvature-aided incremental aggregated gradient method

Wai

Shi

Nedić

et al. 2017

2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

This paper studies an acceleration technique for incremental aggregated gradient methods which exploits curvature information for solving strongly convex finite sum optimization problems. These optimization problems of interest arise in large-scale learning applications relevant to machine learning systems. The proposed methods utilizes a novel curvature-aided gradient tracking technique to produce gradient estimates using the aids of Hessian information during computation. We propose and analyze two curvature-aided methods -the first method, called curvature-aided incremental aggregated gradient (CIAG) method, can be developed from the standard gradient method and it computes an ǫ-optimal solution using O(κ log(1/ǫ)) iterations for a small ǫ; the second method, called accelerated CIAG (A-CIAG) method, incorporates Nesterov's acceleration into CIAG and requires O( √ κ log(1/ǫ)) iterations for a small ǫ, where κ is the problem's condition number. Importantly, the asymptotic convergence rates above are the same as those of the full gradient and accelerated full gradient methods, respectively, and they are independent of the number of component functions involved. The proposed methods are significantly faster than the state-of-the-art methods, especially for large-scale problems with a massive amount of data.

show abstract

Section: Discussionmentioning

confidence: 97%

Section: Gradient Tracking In Incrementalmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Curvature-aided incremental aggregated gradient method

Wai

Shi

Nedić

et al. 2017

2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)

View full text Add to dashboard Cite

show abstract

“…where λ k is the stepsize on each iteration. Proximal gradient method and its variants [13,14,15,16,17,18] have been one hot topic in optimization field for a long time due to their simple forms and lower computation complexity.…”

Section: Introductionmentioning

confidence: 99%

Bregman Proximal Gradient Algorithm With Extrapolation for a Class of Nonconvex Nonsmooth Minimization Problems

et al. 2019

View full text Add to dashboard Cite

In this paper, we consider an accelerated method for solving nonconvex and nonsmooth minimization problems. We propose a Bregman Proximal Gradient algorithm with extrapolation(BPGe). This algorithm extends and accelerates the Bregman Proximal Gradient algorithm (BPG), which circumvents the restrictive global Lipschitz gradient continuity assumption needed in Proximal Gradient algorithms (PG). The BPGe algorithm has higher generality than the recently introduced Proximal Gradient algorithm with extrapolation(PGe), and besides, due to the extrapolation step, BPGe converges faster than BPG algorithm. Analyzing the convergence, we prove that any limit point of the sequence generated by BPGe is a stationary point of the problem by choosing parameters properly. Besides, assuming Kurdyka-Lojasiewicz property, we prove the whole sequences generated by BPGe converges to a stationary point. Finally, to illustrate the potential of the new method BPGe, we apply it to two important practical problems that arise in many fundamental applications (and that not satisfy global Lipschitz gradient continuity assumption): Poisson linear inverse problems and quadratic inverse problems. In the tests the accelerated BPGe algorithm shows faster convergence results, giving an interesting new algorithm.

show abstract

“…However, the linear convergence constants of these methods are not necessarily better than the linear convergence constant of GD for a problem with comparable condition number. This leaves open the possibility that the worst case performance of these methods is worse than the worst case performance of GD -see Section 2. Given that the only difference between stochastic and incremental methods is that in the latter functions are chosen in a cyclic order -as opposed from the selection in stochastic methods which is uniformly at random -it is not surprising that analogous statements can be made for incremental gradient descent methods (IGD) [1,33,22,24,3,25,13,2,34,11,35]. Standard IGD has a slow sublinear convergence rate, which motivates the introduction of memory.…”

mentioning

confidence: 99%

Surpassing Gradient Descent Provably: A Cyclic Incremental Method with Linear Convergence Rate

2018

View full text Add to dashboard Cite

Recently, there has been growing interest in developing optimization methods for solving large-scale machine learning problems. Most of these problems boil down to the problem of minimizing an average of a finite set of smooth and strongly convex functions where the number of functions n is large. Gradient descent method (GD) is successful in minimizing convex problems at a fast linear rate; however, it is not applicable to the considered large-scale optimization setting because of the high computational complexity. Incremental methods resolve this drawback of gradient methods by replacing the required gradient for the descent direction with an incremental gradient approximation. They operate by evaluating one gradient per iteration and executing the average of the n available gradients as a gradient approximate. Although, incremental methods reduce the computational cost of GD, their convergence rates do not justify their advantage relative to GD in terms of the total number of gradient evaluations until convergence. In this paper, we introduce a Double Incremental Aggregated Gradient method (DIAG) that computes the gradient of only one function at each iteration, which is chosen based on a cyclic scheme, and uses the aggregated average gradient of all the functions to approximate the full gradient. The iterates of the proposed DIAG method uses averages of both iterates and gradients in oppose to classic incremental methods that utilize gradient averages but do not utilize iterate averages. We prove that not only the proposed DIAG method converges linearly to the optimal solution, but also its linear convergence factor justifies the advantage of incremental methods on GD. In particular, we prove that the worst case performance of DIAG is better than the worst case performance of GD. Numerical experiments on quadratic programming and logistic regression problems showcase the advantage of DIAG relative to GD and other incremental methods.

show abstract

Global Convergence Rate of Proximal Incremental Aggregated Gradient Methods

Cited by 36 publications

References 31 publications

Curvature-aided incremental aggregated gradient method

Curvature-aided incremental aggregated gradient method

Bregman Proximal Gradient Algorithm With Extrapolation for a Class of Nonconvex Nonsmooth Minimization Problems

Surpassing Gradient Descent Provably: A Cyclic Incremental Method with Linear Convergence Rate

Contact Info

Product

Resources

About