Subsampled Nonmonotone Spectral Gradient Methods

Abstract

This paper deals with subsampled spectral gradient methods for minimizing finite sums. Subsample function and gradient approximations are employed in order to reduce the overall computational cost of the classical spectral gradient methods. The global convergence is enforced by a nonmonotone line search procedure. Global convergence is proved provided that functions and gradients are approximated with increasing accuracy. R-linear convergence and worst-case iteration complexity is investigated in case of strongly convex objective function. Numerical results on well known binary classification problems are given to show the effectiveness of this framework and analyze the effect of different spectral coefficient approximations arising from the variable sample nature of this procedure.

Spectral Projected Subgradient Method for Nonsmooth Convex Optimization Problems

“…This allows us to apply a method of the Spectral Projected Gradient type. The Spectral Projected Gradient (SPG) method, originally proposed in [5], is well known for its efficiency and simplicity and it has been widely used and developed as a solver of constrained optimization problems [4], [8], [11], [16]. The step length selection strategy in SPG method is crucial for faster convergence with respect to classical gradient projection methods because it involves second-order information related to the spectrum of the Hessian matrix.…”

“…The step length selection strategy in SPG method is crucial for faster convergence with respect to classical gradient projection methods because it involves second-order information related to the spectrum of the Hessian matrix. SPG methods for finite sums problem have been investigated in [4], [16]. In [16] they are used in combination with the stochastic gradient method and the convergence is proved assuming that the full gradient is calculated in every m iterations.…”

“…In [16] they are used in combination with the stochastic gradient method and the convergence is proved assuming that the full gradient is calculated in every m iterations. In [4] the subsampled spectral gradient methods are analyzed and the effect of the choice of the spectral coefficient is investigated. In [8] the SPG direction is employed within Inexact Restoration framework to address nonlinear optimization problems with nonconvex constraints.…”

“…This reduces the influence of a noise and usually provides better approximation of the spectral coefficient of the true objective function, but it requires additional evaluations. Although the choice of y k was addressed in the literature (see [4] for example), in general it remains an open question which requires thorough analysis before drawing the final conclusions. It is important to point out that the choice of y k does not affect the convergence analysis and the theoretical results obtained in next section, but it may affect the algorithm's performance significantly.…”