New adaptive stepsize selections in gradient methods

Frassoldati, Giacomo; Zanni, Luca; Zanghirati, Gaetano

doi:10.3934/jimo.2008.4.299

Cited by 126 publications

(157 citation statements)

References 21 publications

(37 reference statements)

Supporting

Mentioning

150

Contrasting

Unclassified

Order By: Relevance

“…In other words, the BB stepsize will approximate all the reciprocals of eigenvalues during the iteration process. Similar observations have been presented in [23]. Thus, for quadratic problem (1) we may consider the following two variants of (18), which Algorithm 1 Gradient method for bound constrained minimization 1: Initialization:…”

Section: (27)mentioning

confidence: 89%

“…Two variants of gradient projection methods combining with the BB stepsizes are also proposed. Our numerical comparisons with DY (6), ABB min 2 [23] and SDC (7) methods on minimizing quadratic functions indicate the proposed strategies and methods are very effective. Moreover, our numerical comparisons with the spectral projected gradient (SPG) method [3,4] on solving bound constrained optimization problems from the CUTEst collection [26] also highly suggest the potential benefits of extending the strategies and methods in the paper for more general large-scale bound constrained optimization.…”

Section: Introductionmentioning

confidence: 93%

“…Firstly, we compare our methods (17) and (18) with the DY method (6) in [14], the ABB min 2 method in [23], and the SDC method (7) in [16] for minimizing quadratic problems. Note that the SDC method has been shown performing better than its monotone variants [16].…”

Section: Quadratic Problemsmentioning

confidence: 99%

“…Based on the observation from Figures 1 and 2, we tested h with values 10 and 20 for our methods. As in [23], the parameter τ of the ABB min 2 method was set to 0.9 for all the problems.…”

Section: Quadratic Problemsmentioning

confidence: 99%

“…The recent study [17] points out that gradient methods using long and short stepsizes that attempt to exploit the spectral properties have generally better numerical performance for minimizing both quadratic and general nonlinear objective functions. For more works on gradient methods, see [7,9,16,17,23,25,37,39]. In [12], Dai and Yang introduced a gradient method with a new stepsize that possesses similar spectral property as α Y k .…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Gradient methods exploiting spectral properties

Huang

Dai

et al. 2020

Optimization Methods and Software

View full text Add to dashboard Cite

We propose a new stepsize for the gradient method. It is shown that this new stepsize will converge to the reciprocal of the largest eigenvalue of the Hessian, when Dai-Yang's asymptotic optimal gradient method (Computational Optimization and Applications, 2006, 33(1): 73-88) is applied for minimizing quadratic objective functions. Based on this spectral property, we develop a monotone gradient method that takes a certain number of steps using the asymptotically optimal stepsize by Dai and Yang, and then follows by some short steps associated with this new stepsize. By employing one step retard of the asymptotic optimal stepsize, a nonmonotone variant of this method is also proposed. Under mild conditions, R-linear convergence of the proposed methods is established for minimizing quadratic functions. In addition, by combining gradient projection techniques and adaptive nonmonotone line search, we further extend those methods for general bound constrained optimization. Two variants of gradient projection methods combining with the Barzilai-Borwein stepsizes are also proposed. Our numerical experiments on both quadratic and bound constrained optimization indicate that the new proposed strategies and methods are very effective.

show abstract

Section: (27)mentioning

confidence: 89%

Section: Introductionmentioning

confidence: 93%

Section: Quadratic Problemsmentioning

confidence: 99%

“…Based on the observation from Figures 1 and 2, we tested h with values 10 and 20 for our methods. As in [23], the parameter τ of the ABB min 2 method was set to 0.9 for all the problems.…”

Section: Quadratic Problemsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Gradient methods exploiting spectral properties

Huang

Dai

et al. 2020

Optimization Methods and Software

View full text Add to dashboard Cite

show abstract

Computing the matrix geometric mean: Riemannian versus Euclidean conditioning, implementation techniques, and a Riemannian BFGS method

Yuan

Huang

Absil

et al. 2020

Numerical Linear Algebra App

View full text Add to dashboard Cite

Summary This paper addresses the problem of computing the Riemannian center of mass of a collection of symmetric positive definite matrices. We show in detail that the condition number of the Riemannian Hessian of the underlying optimization problem is never very ill conditioned in practice, which explains why the Riemannian steepest descent approach has been observed to perform well. We also show theoretically and empirically that this property is not shared by the Euclidean Hessian. We then present a limited‐memory Riemannian BFGS method to handle this computational task. We also provide methods to produce efficient numerical representations of geometric objects that are required for Riemannian optimization methods on the manifold of symmetric positive definite matrices. Through empirical results and a computational complexity analysis, we demonstrate the robust behavior of the limited‐memory Riemannian BFGS method and the efficiency of our implementation when compared to state‐of‐the‐art algorithms.

show abstract

A stochastic variance reduced gradient method with adaptive step for stochastic optimization

Li,

Xue,

Liu

et al. 2024

Optim Control Appl Methods

View full text Add to dashboard Cite

In this paper, we propose a stochastic variance reduction gradient method with adaptive step size, referred to as the SVRG‐New BB method, to solve the convex stochastic optimization problem. The method could be roughly viewed as a hybrid of the SVRG algorithm and a new BB step mechanism. Under the condition that the objective function is strongly convex, we provide the linear convergence proof of this algorithm. Numerical experiment results show that the performance of the SVRG‐New BB algorithm can surpass other existing algorithms if parameters in the algorithm are properly chosen.

show abstract

New adaptive stepsize selections in gradient methods

Cited by 126 publications

References 21 publications

Gradient methods exploiting spectral properties

Gradient methods exploiting spectral properties

Computing the matrix geometric mean: Riemannian versus Euclidean conditioning, implementation techniques, and a Riemannian BFGS method

A stochastic variance reduced gradient method with adaptive step for stochastic optimization

Contact Info

Product

Resources

About