On the Goldstein-Levitin-Polyak gradient projection method

Abstract

This paper considers some aspects of a gradient projection method proposed by Goldstein [l], Levitin and Polyak [3], and more recently, in a less general context, by McCormick [lo]. We propose and analyze some convergent step-size rules to be used in conjunction with the method. These rules are similar in spirit to the efficient Armijo rule for the method of steepest descent and under mild assumptions they have the desirable property that they identify the set of active inequality consbaints in a f ~ t e number of iterations. As a resnlt the method may be converted towards the end of the process to a conjugate direction, quasi-Newton or Newton's method, and achieve the attendant superlinear convergence rate. As an example we propose some quadratically convergent combinations of the method with Newton's method. Such combined methods appear to be very efficient for large-de problems with m a n y simple constraints such as those often appearing in optimal control.

Inexact spectral projected gradient methods on convex sets

“…Letd k be the minimizer of (3). (This minimizer exists and is unique by the strict convexity of the subproblem (3), but we will see later that we do not need to compute it.)…”

“…The SPG method is related to the practical version of Bertsekas [3] of the classical gradient projected method of Goldstein, Levitin and Polyak [21,25]. However, some critical differences make this method much more efficient than its gradient-projection predecessors.…”

Sample size selection in optimization methods for machine learning

“…The last term on the right hand side, 5) measures the error in the Hessian approximation, along the direction d, due to the use of a smaller sample H k . It is apparent from (5.4) that it is inefficient to require that the residual r k be significantly smaller than the Hessian approximation error ∆ H k (w k ; d), as the extra effort in solving the linear system may not lead to an improved search direction for the objective function J S k (w).…”

Variational Hidden Conditional Random Fields with Coupled Dirichlet Process Mixtures

“…However, there is a significant difference with other CRF and HCRF models that use such techniques to find optimal parameters: we are constrained to only positive θ-parameters Since we are using a quasi-Newton method with Armijo backtracking line search, we can use the gradient projection method of [12,13] to enforce this constrain. Finally, it is important to stress here that, although our model includes parameters that are not treated probabilistically, we have not seen signs of overfitting in our experiments (see Fig.…”