Random Barzilai–Borwein step size for mini-batch algorithms

Yang, Zhan; Wang, Cheng; Zhang, Zhemin; Li, Jonathan

doi:10.1016/j.engappai.2018.03.017

Cited by 22 publications

(22 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…showing the effectiveness of our approach. Spectral gradient methods for problem (1) have been investigated in [26,27].…”

Section: Introductionmentioning

confidence: 99%

“…In [26] they are used in combination with the stochastic gradient method, while in [27] a mini-batch strategy is employed. In both papers convergence is proved assuming to employ the full gradient every m iterations (at each outer iteration).…”

Section: Introductionmentioning

confidence: 99%

“…We differ from this latter approach as we embed the variable sample spectral method in a nonmonotone line search strategy using approximate functions and gradients. In the convergence analysis we have also to take into account inaccuracy in function values and not only in gradient, while in [26,27] only inaccuracy in gradients needs to be handled as the function values are unused. On the other hand, the employment of the line search procedure allow us to obtain global convergence to a stationary point irrespectively of convexity of the objective function, differently from [26,27].…”

Section: Introductionmentioning

confidence: 99%

“…In the convergence analysis we have also to take into account inaccuracy in function values and not only in gradient, while in [26,27] only inaccuracy in gradients needs to be handled as the function values are unused. On the other hand, the employment of the line search procedure allow us to obtain global convergence to a stationary point irrespectively of convexity of the objective function, differently from [26,27]. Nonconvex problems arise in neural networks training, so this relaxation can be of great interest.…”

Section: Introductionmentioning

confidence: 99%

“…Notice that in [26] the spectral coefficient is computed at the outer iterations using the full gradient and it is taken constant along the inner iterations. In [27] it is updated also in the inner iterations using a subsample of the dataset whose dimension depends on the condition number of the problem. In [20] spectral gradient methods for problems with objective function given in form of mathematical expectation have been analyzed, however the effect of the choice of the spectral coefficient has not been investigated.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Subsampled Nonmonotone Spectral Gradient Methods

Bellavia

Jerinkic

Malaspina

2020

Communications in Applied and Industrial Mathematics

View full text Add to dashboard Cite

This paper deals with subsampled spectral gradient methods for minimizing finite sums. Subsample function and gradient approximations are employed in order to reduce the overall computational cost of the classical spectral gradient methods. The global convergence is enforced by a nonmonotone line search procedure. Global convergence is proved provided that functions and gradients are approximated with increasing accuracy. R-linear convergence and worst-case iteration complexity is investigated in case of strongly convex objective function. Numerical results on well known binary classification problems are given to show the effectiveness of this framework and analyze the effect of different spectral coefficient approximations arising from the variable sample nature of this procedure.

show abstract

“…showing the effectiveness of our approach. Spectral gradient methods for problem (1) have been investigated in [26,27].…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Subsampled Nonmonotone Spectral Gradient Methods

Bellavia

Jerinkic

Malaspina

2020

Communications in Applied and Industrial Mathematics

View full text Add to dashboard Cite

show abstract

SAAGs: Biased stochastic variance reduction methods for large-scale learning

2019

View full text Add to dashboard Cite

Stochastic approximation is one of the effective approach to deal with the large-scale machine learning problems and the recent research has focused on reduction of variance, caused by the noisy approximations of the gradients. In this paper, we have proposed novel variants of SAAG-I and II (Stochastic Average Adjusted Gradient) Chauhan et el. (2017) [6], called SAAG-III and IV, respectively. Unlike SAAG-I, starting point is set to average of previous epoch in SAAG-III, and unlike SAAG-II, the snap point and starting point are set to average and last iterate of previous epoch in SAAG-IV, respectively. To determine the step size, we have used Stochastic Backtracking-Armijo line Search (SBAS) which performs line search only on selected mini-batch of data points. Since backtracking line search is not suitable for large-scale problems and the constants used to find the step size, like Lipschitz constant, are not always available so SBAS could be very effective in such cases. We have extended SAAGs (I, II, III and IV) to solve non-smooth problems and designed two update rules for smooth and nonsmooth problems. Moreover, our theoretical results have proved linear convergence

show abstract

Stochastic variance reduced gradient with hyper-gradient for non-convex large-scale learning

Yang

2023

Appl Intell

View full text Add to dashboard Cite

Random Barzilai–Borwein step size for mini-batch algorithms

Cited by 22 publications

References 28 publications

Subsampled Nonmonotone Spectral Gradient Methods

Subsampled Nonmonotone Spectral Gradient Methods

SAAGs: Biased stochastic variance reduction methods for large-scale learning

Stochastic variance reduced gradient with hyper-gradient for non-convex large-scale learning

Contact Info

Product

Resources

About