Gradient projection methods for quadratic programs and applications in training support vector machines

Abstract

Gradient projection methods based on the Barzilai-Borwein spectral steplength choices are considered for quadratic programming (QP) problems with simple constraints. Well-known nonmonotone spectral projected gradient methods and variable projection methods are discussed. For both approaches, the behavior of different combinations of the two spectral steplengths is investigated. A new adaptive steplength alternating rule is proposed, which becomes the basis for a generalized version of the variable projection method (GVPM). Convergence results are given for the proposed approach and its effectiveness is shown by means of an extensive computational study on several test problems, including the special quadratic programs arising in training support vector machines (SVMs). Finally, the GVPM behavior as inner QP solver in decomposition techniques for large-scale SVMs is also evaluated

The cyclic Barzilai-–Borwein method for unconstrained optimization

“…Other possible choices for the stepsize α k include Dai (2003), Dai & Fletcher (2006), Dai & Yang (2001, 2003, Friedlander et al (1999), Grippo & Sciandrone (2002), Raydan & Svaiter (2002) and Serafini et al (2005). In this paper, we refer to (1.6) as the BB formula.…”

“…Liu & Dai (2001) provide a powerful scheme for solving noisy unconstrained optimization problems by combining the BB method and a stochastic approximation method. The projected BBlike method turns out to be very useful in machine learning for training support vector machines (see Serafini et al, 2005;Dai & Fletcher, 2006). Empirically, good performance is observed on a wide variety of classification problems.…”

Duality-based algorithms for total-variation-regularized image restoration

“…Numerous variants have been proposed recently, and subjected to with theoretical and computational analysis. The BB step-length rules have also been extended to constrained optimization, particularly to bound-constrained quadratic programming; see, for example [10] and [18]. The same formulae (24) and (25) can be used in these cases to determine the step length, but we obtain x k+1 by projecting x k − α k ∇F (x k ) onto the feasible set X, and possibly performing additional backtracking or line-search modifications to ensure descent in F .…”

Decomposition Algorithms for Training Large-Scale Semiparametric Support Vector Machines

“…We implemented both the decomposition framework (Algorithm 1) and the PDSG subproblem solver (Algorithm 2) in C++. The code was developed by modifying the GPDT code of Serafini, Zanghirati, and Zanni [5] 1 , and retains many features of this code. Our code caches once-computed kernel entries for reuse, with the least-recently-used (LRU) replacement strategy.…”

“…Although we face the burden of designing an efficient, robust QP solver, methods in the second group often show faster convergence than those in the first group. Successful instances of methods in the second group include SVM light [4] and GPDT [5,6]. The QP solvers used in the second group can be applied to the full problem, thus solving it in one "outer" iteration, though this approach is not usually effective for large data sets.…”