On the working set selection in gradient projection-based decomposition techniques for support vector machines

Abstract

This work deals with special decomposition techniques for the large quadratic program arising in training Support Vector Machines. These approaches split the problem into a sequence of quadratic programming subproblems which can be solved by efficient gradient projection methods recently proposed. By decomposing into much larger subproblems than standard decomposition packages, these techniques show promising performance and are well suited for parallelization. Here, we discuss a crucial aspect for their effectiveness: the selection of the working set, that is the index set of the variables to be optimized at each step through the quadratic programming subproblem. We analyse the most popular working set selections and develop a new selection strategy that improves the convergence rate of the decomposition schemes based on large sized working sets. The effectiveness of the proposed strategy within the gradient projectionbased decomposition techniques is shown by numerical experiments on large benchmark problems, both in serial and parallel environments.

Decomposition Algorithms for Training Large-Scale Semiparametric Support Vector Machines

“…We choose the indices of the off-bounds components (0 < x k+1 i < C) first, and then those of lower and upper bounds. We reduce n c as the change between two consecutive working sets decreases, as in [6]. We observe that adaptive reduction of n c provides better convergence of the Lagrange multiplier η k , and helps avoid zigzagging between two working sets without making further progress.…”

“…Our approach is motivated by the KKT conditions (4), and indeed can be solved by simply sorting the violations of these conditions. It contrasts with previous methods [4,6,12], in which the equality constraints are enforced explicitly in the working set selection subproblem. Our approach has no requirements on the size of n c , yet it is still effective when η k+1 is close to the optimal value η * .…”

“…The selection of working set B at each outer iteration is inspired by the approach of Joachims [4], later improved by Serafini and Zanni [6]. The size of the working set is fixed at some value n B , of which up to n c are allowed to be "fresh" indices while the remainder are carried over from the current working set.…”

“…Although we face the burden of designing an efficient, robust QP solver, methods in the second group often show faster convergence than those in the first group. Successful instances of methods in the second group include SVM light [4] and GPDT [5,6]. The QP solvers used in the second group can be applied to the full problem, thus solving it in one "outer" iteration, though this approach is not usually effective for large data sets.…”

A Convergent Hybrid Decomposition Algorithm Model for SVM Training

“…[13]), decomposition methods (see, e.g., Refs. [14][15][16][17]). Here we focus our attention on a decomposition-based approach which, involving at each iteration the updating of a small number of variables, is suitable for large problems with dense Hessian matrix.…”