Pegasos: primal estimated sub-gradient solver for SVM

Abstract

We describe and analyze a simple and effective stochastic sub-gradient descent algorithm for solving the optimization problem cast by Support Vector Machines (SVM). We prove that the number of iterations required to obtain a solution of accuracy isÕ(1/ ), where each iteration operates on a single training example. In contrast, previous analyses of stochastic gradient descent methods for SVMs require Ω(1/ 2 ) iterations. As in previously devised SVM solvers, the number of iterations also scales linearly with 1/λ, where λ is the regularization parameter of SVM. For a linear kernel, the total run-time of our method isÕ(d/(λ )), where d is a bound on the number of non-zero features in each example. Since the run-time does not depend directly on the size of the training set, the resulting algorithm is especially suited for learning from large datasets. Our approach also extends to non-linear kernels while working solely on the primal objective function, though in this case the runtime does depend linearly on the training set size. Our algorithm is particularly well suited for large text classification problems, where we demonstrate an order-of-magnitude speedup over previous SVM learning methods.

Two New Decomposition Algorithms for Training Bound-Constrained Support Vector Machines*

“…It can be checked easily that the above chosen u and v, together with α satisfy the KKT condition (17).…”

“…The existing algorithms can be broadly categorized into two categories: the cutting-plane methods [11,5,12,13,25], and subgradient methods [3,17]. For example, in [17], Shalev-Shwartz et al described and analyzed a simple and effective stochastic sub-gradient descent algorithm and prove that the number of iterations required to obtain a solution of accuracy is O(1/ ). Generally speaking, without counting the loading time, these recent advances on linear classification have shown that training one million instances takes only a few seconds [22].…”

A Latent Variable Ranking Model for Content-Based Retrieval

“…We used a primal projected sub-gradient algorithm [19], and tuned the regularization constants on the validation data. The mixture model took typically less than 10 global iterations to converge.…”

“…This corresponds to a ranking SVM [11]. One simple strategy to minimize this objective is to use a primal sub-gradient method [19], which is the approach we use in this paper.…”

Approximately Minwise Independence with Twisted Tabulation

“…However, the problem with bottom-q sketches, is that the samples lose their alignment. In applications of large-scale machine learning this alignment is needed in order to efficiently construct a dot-product for use with a linear support vector machine (SVM) 4 such as LIBLINEAR [9] or Pegasos [19]. Using the alignment of kˆminwise, it was shown how to construct such a dot-product in [13] based on this scheme.…”