Non-monotone Behavior of the Heavy Ball Method

Danilova, Marina; Кулакова, Анастасия Игоревна; Polyak, B. T.

doi:10.1007/978-3-030-35502-9_9

Cited by 7 publications

(5 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the use of proper values of β 2 , we can realize the damping of oscillations related to the non-monotonic convergence of the method. This is typical for the case of κ ≫ 1 [13]. In Section 3, this will also be illustrated for the minimization of non-quadratic convex and non-convex functions.…”

Section: Equivalent Odementioning

confidence: 96%

“…This ODE describes the descent process in the neighborhood of x * . As can be seen from ( 25), the presence of β 2 realized an additional damping of oscillations associated with non-monotone convergence of the HBM [13]. 4.…”

mentioning

confidence: 93%

“…Recently, Goujand et al [12] proposed an adaptive modification of the HBM with Polyak stepsizes and demonstrated that this method can be considered a variant of the conjugate gradient method for quadratic problems, having many advantages, such as finite-time convergence and instant optimality. Danilova et al [13] demonstrated the non-monotonic convergence of the HBM and analyzed the peak effect for ill-conditioned problems. In order to carry out the damping of this effect in [14], an averaged HBM was constructed.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Analysis of a Two-Step Gradient Method with Two Momentum Parameters for Strongly Convex Unconstrained Optimization

Krivovichev,

Sergeeva

2024

Algorithms

View full text Add to dashboard Cite

The paper is devoted to the theoretical and numerical analysis of the two-step method, constructed as a modification of Polyak’s heavy ball method with the inclusion of an additional momentum parameter. For the quadratic case, the convergence conditions are obtained with the use of the first Lyapunov method. For the non-quadratic case, sufficiently smooth strongly convex functions are obtained, and these conditions guarantee local convergence.An approach to finding optimal parameter values based on the solution of a constrained optimization problem is proposed. The effect of an additional parameter on the convergence rate is analyzed. With the use of an ordinary differential equation, equivalent to the method, the damping effect of this parameter on the oscillations, which is typical for the non-monotonic convergence of the heavy ball method, is demonstrated. In different numerical examples for non-quadratic convex and non-convex test functions and machine learning problems (regularized smoothed elastic net regression, logistic regression, and recurrent neural network training), the positive influence of an additional parameter value on the convergence process is demonstrated.

show abstract

Section: Equivalent Odementioning

confidence: 96%

mentioning

confidence: 93%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of a Two-Step Gradient Method with Two Momentum Parameters for Strongly Convex Unconstrained Optimization

Krivovichev,

Sergeeva

2024

Algorithms

View full text Add to dashboard Cite

show abstract

“…for all k ≥ 1. Our choice of θ k differs from the existing ones; the existing complexity analyses [16,17,21,32,34,43] of HB prohibit θ k = 1. For example, Li and Lin [34] proposed…”

Section: Update Of Solutionsmentioning

confidence: 99%

Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians

Marumo,

Takeda

2024

Math. Program.

View full text Add to dashboard Cite

We propose a new first-order method for minimizing nonconvex functions with Lipschitz continuous gradients and Hölder continuous Hessians. The proposed algorithm is a heavy-ball method equipped with two particular restart mechanisms. It finds a solution where the gradient norm is less than $$\varepsilon $$ ε in $$O(H_{\nu }^{\frac{1}{2 + 2 \nu }} \varepsilon ^{- \frac{4 + 3 \nu }{2 + 2 \nu }})$$ O ( H ν 1 2 + 2 ν ε - 4 + 3 ν 2 + 2 ν ) function and gradient evaluations, where $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] and $$H_{\nu }$$ H ν are the Hölder exponent and constant, respectively. This complexity result covers the classical bound of $$O(\varepsilon ^{-2})$$ O ( ε - 2 ) for $$\nu = 0$$ ν = 0 and the state-of-the-art bound of $$O(\varepsilon ^{-7/4})$$ O ( ε - 7 / 4 ) for $$\nu = 1$$ ν = 1 . Our algorithm is $$\nu $$ ν -independent and thus universal; it automatically achieves the above complexity bound with the optimal $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] without knowledge of $$H_{\nu }$$ H ν . In addition, the algorithm does not require other problem-dependent parameters as input, including the gradient’s Lipschitz constant or the target accuracy $$\varepsilon $$ ε . Numerical results illustrate that the proposed method is promising.

show abstract

“…Actually, fast error/noise accumulation is a typical drawback of accelerated SGD with small batchsizes [35]. Moreover, deterministic accelerated and momentum-based methods often have non-monotone behavior (see [5] and references therein). However, to some extent clipped-SSTM suffers from the first drawback less than SSTM and has comparable convergence rate with SSTM.…”

Section: Numerical Experimentsmentioning

confidence: 99%

Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping

Gorbunov¹,

Danilova²,

Gasnikov³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a new accelerated stochastic first-order method called clipped-SSTM for smooth convex stochastic optimization with heavy-tailed distributed noise in stochastic gradients and derive the first high-probability complexity bounds for this method closing the gap in the theory of stochastic optimization with heavy-tailed noise. Our method is based on a special variant of accelerated Stochastic Gradient Descent (SGD) and clipping of stochastic gradients. We extend our method to the strongly convex case and prove new complexity bounds that outperform state-of-the-art results in this case. Finally, we extend our proof technique and derive the first non-trivial high-probability complexity bounds for SGD with clipping without light-tails assumption on the noise.

show abstract

Non-monotone Behavior of the Heavy Ball Method

Cited by 7 publications

References 14 publications

Analysis of a Two-Step Gradient Method with Two Momentum Parameters for Strongly Convex Unconstrained Optimization

Analysis of a Two-Step Gradient Method with Two Momentum Parameters for Strongly Convex Unconstrained Optimization

Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians

Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping

Contact Info

Product

Resources

About