Quadruply Stochastic Gradient Method for Large Scale Nonlinear Semi-Supervised Ordinal Regression AUC Optimization

Shi, Wanli; Gu, Bin; Li, Xiang; Huang, Heng

doi:10.1609/aaai.v34i04.6029

Cited by 11 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hitherto, a substantial amount of efforts have been made to explore the AUC optimization method/theory for the multipartite ranking problem [11], [13], [24], [63], [69]. Moreover, a recent work [66] proposes a novel nonlinear semi-supervised multipartite ranking problem for large-scale datasets. It is noteworthy that multipartite ranking approaches could solve multiclass problems only if the classes are ordinal values for the same semantic concept.…”

Section: Related Workmentioning

confidence: 99%

Learning With Multiclass AUC: Theory and Algorithms

Yang

Bao

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

The Area under the ROC curve (AUC) is a well-known ranking metric for problems such as imbalanced learning and recommender systems. The vast majority of existing AUC-optimization-based machine learning methods only focus on binary-class cases, while leaving the multiclass cases unconsidered. In this paper, we start an early trial to consider the problem of learning multiclass scoring functions via optimizing multiclass AUC metrics. Our foundation is based on the M metric, which is a wellknown multiclass extension of AUC. We first pay a revisit to this metric, showing that it could eliminate the imbalance issue from the minority class pairs. Motivated by this, we propose an empirical surrogate risk minimization framework to approximately optimize the M metric. Theoretically, we show that: (i) optimizing most of the popular differentiable surrogate losses suffices to reach the Bayes optimal scoring function asymptotically; (ii) the training framework enjoys an imbalance-aware generalization error bound, which pays more attention to the bottleneck samples of minority classes compared with the traditional O( 1/N ) result. Practically, to deal with the low scalability of the computational operations, we propose acceleration methods for three popular surrogate loss functions, including the exponential loss, squared loss, and hinge loss, to speed up loss and gradient evaluations. Finally, experimental results on 11 real-world datasets demonstrate the effectiveness of our proposed framework.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning With Multiclass AUC: Theory and Algorithms

Yang

Bao

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…A convergence rate in the order O ( 1𝑇 ) for the optimization error was established for the strongly regularized empirical AUC maximization problem. The work [131] also considers a similar algorithm for semi-supervised ordinal regression based on AUC optimization.…”

Section: Stochastic Auc Maximization -The Third Agementioning

confidence: 99%

AUC Maximization in the Era of Big Data and AI: A Survey

Yang¹,

Ying²

2022

Preprint

View full text Add to dashboard Cite

Area under the ROC curve, a.k.a. AUC, is a measure of choice for assessing the performance of a classifier for imbalanced data. AUC maximization refers to a learning paradigm that learns a predictive model by directly maximizing its AUC score. It has been studied for more than two decades dating back to late 90s and a huge amount of work has been devoted to AUC maximization since then. Recently, stochastic AUC maximization for big data and deep AUC maximization for deep learning have received increasing attention and yielded great impact for solving real-world problems. However, to the best our knowledge there is no comprehensive survey of related works for AUC maximization. This paper aims to address the gap by reviewing the literature in the past two decades. We not only give a holistic view of the literature but also present detailed explanations and comparisons of different papers from formulations to algorithms and theoretical guarantees. We also identify and discuss remaining and emerging issues for deep AUC maximization, and provide suggestions on topics for future work.

show abstract

“…As a result, f t+1 may be outside of RKHS H, making it hard to directly evaluate the error between f t+1 (•) and the optimal solution f * . In this case, we utilize h t+1 (•) as an intermediate value to decompose the difference between f t+1 and f * (Shi et al 2020):…”

Section: Convergence Analysismentioning

confidence: 99%

Fast and Scalable Adversarial Training of Kernel SVM via Doubly Stochastic Gradients

2021

AAAI

Self Cite

View full text Add to dashboard Cite

Adversarial attacks by generating examples which are almost indistinguishable from natural examples, pose a serious threat to learning models. Defending against adversarial attacks is a critical element for a reliable learning system. Support vector machine (SVM) is a classical yet still important learning algorithm even in the current deep learning era. Although a wide range of researches have been done in recent years to improve the adversarial robustness of learning models, but most of them are limited to deep neural networks (DNNs) and the work for kernel SVM is still vacant. In this paper, we aim at kernel SVM and propose adv-SVM to improve its adversarial robustness via adversarial training, which has been demonstrated to be the most promising defense techniques. To the best of our knowledge, this is the first work that devotes to the fast and scalable adversarial training of kernel SVM. Specifically, we first build connection of perturbations of samples between original and kernel spaces, and then give a reduced and equivalent formulation of adversarial training of kernel SVM based on the connection. Next, doubly stochastic gradients (DSG) based on two unbiased stochastic approximations (i.e., one is on training points and another is on random features) are applied to update the solution of our objective function. Finally, we prove that our algorithm optimized by DSG converges to the optimal solution at the rate of O(1/$t$) under the constant and diminishing stepsizes. Comprehensive experimental results show that our adversarial training algorithm enjoys robustness against various attacks and meanwhile has the similar efficiency and scalability with classical DSG algorithm.

show abstract

Quadruply Stochastic Gradient Method for Large Scale Nonlinear Semi-Supervised Ordinal Regression AUC Optimization

Cited by 11 publications

References 21 publications

Learning With Multiclass AUC: Theory and Algorithms

Learning With Multiclass AUC: Theory and Algorithms

AUC Maximization in the Era of Big Data and AI: A Survey

Fast and Scalable Adversarial Training of Kernel SVM via Doubly Stochastic Gradients

Contact Info

Product

Resources

About