Large scale semi-supervised linear SVMs

Abstract

Large scale learning is often realistic only in a semi-supervised setting where a small set of labeled examples is available together with a large collection of unlabeled data. In many information retrieval and data mining applications, linear classifiers are strongly preferred because of their ease of implementation, interpretability and empirical performance. In this work, we present a family of semi-supervised linear support vector classifiers that are designed to handle partially-labeled sparse datasets with possibly very large number of examples and features. At their core, our algorithms employ recently developed modified finite Newton techniques. Our contributions in this paper are as follows: (a) We provide an implementation of Transductive SVM (TSVM) that is significantly more efficient and scalable than currently used dual techniques, for linear classification problems involving large, sparse datasets. (b) We propose a variant of TSVM that involves multiple switching of labels. Experimental results show that this variant provides an order of magnitude further improvement in training efficiency. (c) We present a new algorithm for semi-supervised learning based on a Deterministic Annealing (DA) approach. This algorithm alleviates the problem of local minimum in the TSVM optimization procedure while also being computationally attractive. We conduct an empirical study on several document classification tasks which confirms the value of our methods in large scale semisupervised settings.

A User’s Guide to Support Vector Machines

“…Researchers have therefore explored ways to achieve faster training times. For linear SVMs very efficient solvers are available which converge in a time which is linear in the number of examples [16,17,15]. Approximate solvers that can be trained in linear time without a significant loss of accuracy were also developed [18].…”

An Optimization Framework for Combining Ensembles of Classifiers and Clusterers with Applications to Nontransductive Semisupervised Learning and Transfer Learning

“…The proofs for the convergence are novel and generalize across a wide variety of Bregman divergences, allowing one to use a suitable divergence measure based on the application domain. The proposed framework has been empirically shown to outperform a variety of algorithms [Gao et al 2008[Gao et al , 2013Sindhwani and Keerthi 2006] in both semisupervised and transfer learning problems. More significantly, it can operate even in settings where there are no labeled data in the target domain, and the labeled data from the source domain are also no longer available.…”

“…Linear Support Vector Machine (S 3 VM) [Sindhwani and Keerthi 2006] for comparison. Finally, in Section 6.5, we report empirical results for transfer learning settings.…”

“…We also compare OAC 3 to a popular semisupervised algorithm known as S 3 VM [Sindhwani and Keerthi 2006]. This algorithm is essentially a Transductive Linear SVM) that can be viewed as a large-scale implementation of the algorithm introduced in Joachims [1999b].…”

“…This algorithm is essentially a Transductive Linear SVM) that can be viewed as a large-scale implementation of the algorithm introduced in Joachims [1999b]. For dealing with unlabeled data, it appends an additional term in the SVM objective function whose role is to drive the classification hyperplane toward low data density regions [Sindhwani and Keerthi 2006]. The default parameter values have been used for S 3 VM.…”