Scalable Semi-Supervised SVM via Triply Stochastic Gradients

Abstract

Semi-supervised learning (SSL) plays an increasingly important role in the big data era because a large number of unlabeled samples can be used effectively to improve the performance of the classifier. Semi-supervised support vector machine (S 3 VM) is one of the most appealing methods for SSL, but scaling up S 3 VM for kernel learning is still an open problem. Recently, a doubly stochastic gradient (DSG) algorithm has been proposed to achieve efficient and scalable training for kernel methods. However, the algorithm and theoretical analysis of DSG are developed based on the convexity assumption which makes them incompetent for non-convex problems such as S 3 VM. To address this problem, in this paper, we propose a triply stochastic gradient algorithm for S 3 VM, called TSGS 3 VM. Specifically, to handle two types of data instances involved in S 3 VM, TSGS 3 VM samples a labeled instance and an unlabeled instance as well with the random features in each iteration to compute a triply stochastic gradient. We use the approximated gradient to update the solution. More importantly, we establish new theoretic analysis for TSGS 3 VM which guarantees that TSGS 3 VM can converge to a stationary point. Extensive experimental results on a variety of datasets demonstrate that TSGS 3 VM is much more efficient and scalable than existing S 3 VM algorithms.

Triply stochastic gradient method for large-scale nonlinear similar unlabeled classification

“…Since our proposed method contains three sources of randomness, we denote our method as Triply Stochastic Gradient descent for SU Classification (TSGSU). Theoretically, we give a new theoretically analysis based on the framework in Geng et al (2019), Dai et al (2014) and prove that our method can converge to the stationary point at the rate of O( 1 √ T ) after T iterations. Our experiments on various benchmark datasets and high-dimensional datasets not only demonstrate the scalability but also show the efficiency of TSGSU compared with existing learning algorithms while retaining similar generalization performance.…”

“…It uses pseudo-random number generators to generate the random features on the fly, which highly reduces the memory requirement of RFF. Due to its superior performance, DSG has been successfully applied to scale up kernelbased algorithms in many applications, such as (Gu et al 2018b;Li et al 2017;Rahimi and Recht 2009;Le et al 2013;Shi et al 2019;Geng et al 2019). The theoretical analysis of Dai et al (2014), Gu et al (2018b), Li et al (2017), Shi et al (2019) are all based on the assumption that the objective functions of these problems are convex.…”

“…However, in many other practical applications, such as disaster resilience, medical diagnosis, and bioinformatics, massive labeled data cannot be collected easily since manually labeling the unlabeled data is time-consuming and laborious. To handle this problem, great efforts have been done in weakly-supervised classification, including semi-supervised learning (Chapelle et al 2009;Sakai et al 2017Sakai et al , 2018Geng et al 2019;Shi et al 2019;Yu et al 2019), positive unlabeled (PU) learning (du Plessis et al 2014(du Plessis et al , 2015b and one-class classification (Khan and Madden 2009;Schölkopf et al 2001).…”

Towards Visual Explainable Active Learning for Zero-Shot Classification

“…However, solving the S 3 V M problem is very challenging for its nonconvexity and computational cost [23]. It is still an open problem to scale up S 3 V M for large-scale applications.…”

Quadruply Stochastic Gradient Method for Large Scale Nonlinear Semi-Supervised Ordinal Regression AUC Optimization

“…However, RFF method needs to save large amounts of random features. Instead of saving all the random features, Dai et al, (2014) proposed DSG algorithm to use pseudorandom number generators to generate the random features on-the-fly, which has been widely used (Shi et al 2019;Geng et al 2019;Li et al 2017). Our method can be viewed as an extension of (Shi et al 2019).…”