Global convergence of the Heavy-ball method for convex optimization

Ghadimi, Euhanna; Feyzmahdavian, Hamid Reza; Johansson, Mikael

doi:10.1109/ecc.2015.7330562

Cited by 199 publications

(197 citation statements)

References 19 publications

Supporting

Mentioning

191

Contrasting

Order By: Relevance

“…Our result improves the linear convergence established by Ghadimi, Feyzmahdavian, and Johansson (2015) in two aspects: Firstly, The strongly convex assumption is weakened to (19). Secondly, The step size and inertial parameter are chosen independent of the strongly convex constants.…”

Section: Linear Convergence With Restricted Strong Convexitysupporting

confidence: 69%

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Sun

Yin

et al. 2019

AAAI

View full text Add to dashboard Cite

In this paper, we revisit the convergence of the Heavyball method, and present improved convergence complexity results in the convex setting. We provide the first non-ergodic O(1/k) rate result of the Heavy-ball algorithm with constant step size for coercive objective functions. For objective functions satisfying a relaxed strongly convex condition, the linear convergence is established under weaker assumptions on the step size and inertial parameter than made in the existing literature. We extend our results to multi-block version of the algorithm with both the cyclic and stochastic update rules. In addition, our results can also be extended to decentralized optimization, where the ergodic analysis is not applicable.

show abstract

Section: Linear Convergence With Restricted Strong Convexitysupporting

confidence: 69%

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Sun

Yin

et al. 2019

AAAI

View full text Add to dashboard Cite

show abstract

“…Global convergence of the Heavy Ball method is established here for function under condition (19) through Lyapunov function (13). The similar results on global convergence for more narrow class of strongly convex functions were obtained in paper [13].…”

Section: Global Convergencesupporting

confidence: 60%

Non-monotone Behavior of the Heavy Ball Method

Danilova

Кулакова

Polyak

2020

Springer Proceedings in Mathematics &Amp; Statistics

View full text Add to dashboard Cite

We focus on the solutions of second-order stable linear difference equations and demonstrate that their behavior can be non-monotone and exhibit peak effects depending on initial conditions. The results are applied to the analysis of the accelerated unconstrained optimization method -the Heavy Ball method. We explain non-standard behavior of the method discovered in practical applications. In addition, such non-monotonicity complicates the correct choice of the parameters in optimization methods. We propose to overcome this difficulty by introducing new Lyapunov function which should decrease monotonically. By use of this function convergence of the method is established under less restrictive assumptions (for instance, with the lack of convexity). We also suggest some restart techniques to speed up the method's convergence.

show abstract

“…Theoretically, it has been proved that HBGD enjoys better convergence factor than both the gradient and Nesterov's accelerated gradient method with linear convergence rates under the condition that the objective function is twice continuously differentiable, strongly convex and has Lipschitz continuous gradient. With the convex and smooth assumption on the objective function, the ergodic O(1/k) rate in terms of the objective value, i.e., f [Ghadimi et al, 2015]. HBGD was proved to converge linearly in the strongly convex case by [Ghadimi et al, 2015].…”

Section: Heavy-ball Algorithmsmentioning

confidence: 99%

“…With the convex and smooth assumption on the objective function, the ergodic O(1/k) rate in terms of the objective value, i.e., f [Ghadimi et al, 2015]. HBGD was proved to converge linearly in the strongly convex case by [Ghadimi et al, 2015]. But the authors used a somehow restrictive assumption on the parameter β, which leads to a small range values of choose for β when the strongly convex constant is tiny.…”

Section: Heavy-ball Algorithmsmentioning

confidence: 99%

Heavy-ball Algorithms Always Escape Saddle Points

Sun

Quan

et al. 2019

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Nonconvex optimization algorithms with random initialization have attracted increasing attention recently. It has been showed that many first-order methods always avoid saddle points with random starting points. In this paper, we answer a question: can the nonconvex heavy-ball algorithms with random initialization avoid saddle points? The answer is yes! Direct using the existing proof technique for the heavy-ball algorithms is hard due to that each iteration of the heavy-ball algorithm consists of current and last points. It is impossible to formulate the algorithms as iteration like x k+1 = g(x k ) under some mapping g. To this end, we design a new mapping on a new space. With some transfers, the heavy-ball algorithm can be interpreted as iterations after this mapping. Theoretically, we prove that heavy-ball gradient descent enjoys larger stepsize than the gradient descent to escape saddle points to escape the saddle point. And the heavyball proximal point algorithm is also considered; we also proved that the algorithm can always escape the saddle point.

show abstract

Global convergence of the Heavy-ball method for convex optimization

Cited by 199 publications

References 19 publications

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Non-Ergodic Convergence Analysis of Heavy-Ball Algorithms

Non-monotone Behavior of the Heavy Ball Method

Heavy-ball Algorithms Always Escape Saddle Points

Contact Info

Product

Resources

About