Quadruply Stochastic Gradients for Large Scale Nonlinear Semi-Supervised AUC Optimization

Abstract

Semi-supervised ordinal regression (S 2 OR) problems are ubiquitous in real-world applications, where only a few ordered instances are labeled and massive instances remain unlabeled. Recent researches have shown that directly optimizing concordance index or AUC can impose a better ranking on the data than optimizing the traditional error rate in ordinal regression (OR) problems. In this paper, we propose an unbiased objective function for S 2 OR AUC optimization based on ordinal binary decomposition approach. Besides, to handle the large-scale kernelized learning problems, we propose a scalable algorithm called QS 3 ORAO using the doubly stochastic gradients (DSG) framework for functional optimization. Theoretically, we prove that our method can converge to the optimal solution at the rate of O(1/t), where t is the number of iterations for stochastic data sampling. Extensive experimental results on various benchmark and realworld datasets also demonstrate that our method is efficient and effective while retaining similar generalization performance.

Scalable Semi-Supervised SVM via Triply Stochastic Gradients

“…There are many kernel approximation methods proposed to address the scalability issue of kernel methods. et al, 2017;Gu et al, 2018c;Shi et al, 2019] can not be used for S 3 VM as discussed previously.…”

“…For the method of self-labeling heuristics, Joachims [1999] proposed a S 3 VM light algorithm which uses self-labeling heuristics for labeling the unlabeled data, then iteratively solve this standard SVM until convergence. CCCP-based methods were proposed to solve S 3 VM in [Chapelle and Zien, 2005;Wang et al, 2007;Yu et al, 2019]. The basic principle of CCCP is to linearize the concave part of S 3 VM's objective function around a solution obtained in the current iteration so that sub-problem is convex.…”

“…Thus, DSG avoids computing and storing a kernel matrix, while enjoying nice computational and space complexities. Xie et al [2015] used DSG to scale up nonlinear component analysis. To the best of our knowledge, [Xie et al, 2015] is the only work based on DSG to solve a non-convex problem.…”

“…However, DSG only considers the expected error on the labeled data. 2) Non-convexity analysis: The theoretical analysis in [Xie et al, 2015] is based on a strong assumption (i.e., the initialization needs to be close to the optimum). However, such an assumption is rarely satisfied in practice.…”

“…Besides, they focus on the nonlinear component analysis instead of general non-convex problems. Thus, it is infeasible to extend the analysis of [Xie et al, 2015] to S 3 VM.…”