Stochastic AUC optimization with general loss

Yang, Zhenhuan; Shen, Wei; Ying, Yiming; Yuan, Xiaoming

doi:10.3934/cpaa.2020188

Cited by 7 publications

(9 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 5. Comparison of different studies for stochastic AUC maximization algorithms, where 𝑇 is the total number of iterations, 𝑑 denotes the dimensionality of the input data, 𝐵 is the batch size used in [167] on which the parameter 𝜌 + 𝑘 (𝐵) is dependent, 𝑚 is the degree of Bernstein polynomials to approximate the general convex loss used in [164], 𝜌 + 𝑘 (𝐵) and 𝜌 − 𝑘 are RSC and RSS parameters used in [167], and 𝛽, 𝑀 and 𝜂 respectively denote the strong-convexity parameter, the smooth parameter and the constant step size in [31]. Opt.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

AUC Maximization in the Era of Big Data and AI: A Survey

Yang¹,

Ying²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Area under the ROC curve, a.k.a. AUC, is a measure of choice for assessing the performance of a classifier for imbalanced data. AUC maximization refers to a learning paradigm that learns a predictive model by directly maximizing its AUC score. It has been studied for more than two decades dating back to late 90s and a huge amount of work has been devoted to AUC maximization since then. Recently, stochastic AUC maximization for big data and deep AUC maximization for deep learning have received increasing attention and yielded great impact for solving real-world problems. However, to the best our knowledge there is no comprehensive survey of related works for AUC maximization. This paper aims to address the gap by reviewing the literature in the past two decades. We not only give a holistic view of the literature but also present detailed explanations and comparisons of different papers from formulations to algorithms and theoretical guarantees. We also identify and discuss remaining and emerging issues for deep AUC maximization, and provide suggestions on topics for future work.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Recently, the work [164] also proposes stochastic primal-dual algorithm for solving AUC maximization with a general convex pairwise loss. They propose to use Bernstein polynomials [117] to uniformly approximate a general loss.…”

Section: Stochastic Auc Maximization -The Third Agementioning

confidence: 99%

AUC Maximization in the Era of Big Data and AI: A Survey

Yang¹,

Ying²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…[59], [60] further accelerate this framework with tighter convergence rates. On top of the reformulation framework, [75] also provides an acceleration framework for general loss functions where the loss functions are approximated by the Bernstein polynomials. [18] proposes a novel large-scale nonlinear AUC maximization method based on the triply stochastic gradient descent algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…Unfortunately, the lossless acceleration for general losses is impossible in general. In this subsection, following the spirit of [75], we provide a discussion on how to accelerate general loss functions approximately. However, we can, in turn, construct an approximation framework based on the Bernstein Polynomials.…”

Section: G4 Acceleration Scheme For General Lossesmentioning

confidence: 99%

See 1 more Smart Citation

Learning With Multiclass AUC: Theory and Algorithms

Yang

Bao

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

The Area under the ROC curve (AUC) is a well-known ranking metric for problems such as imbalanced learning and recommender systems. The vast majority of existing AUC-optimization-based machine learning methods only focus on binary-class cases, while leaving the multiclass cases unconsidered. In this paper, we start an early trial to consider the problem of learning multiclass scoring functions via optimizing multiclass AUC metrics. Our foundation is based on the M metric, which is a wellknown multiclass extension of AUC. We first pay a revisit to this metric, showing that it could eliminate the imbalance issue from the minority class pairs. Motivated by this, we propose an empirical surrogate risk minimization framework to approximately optimize the M metric. Theoretically, we show that: (i) optimizing most of the popular differentiable surrogate losses suffices to reach the Bayes optimal scoring function asymptotically; (ii) the training framework enjoys an imbalance-aware generalization error bound, which pays more attention to the bottleneck samples of minority classes compared with the traditional O( 1/N ) result. Practically, to deal with the low scalability of the computational operations, we propose acceleration methods for three popular surrogate loss functions, including the exponential loss, squared loss, and hinge loss, to speed up loss and gradient evaluations. Finally, experimental results on 11 real-world datasets demonstrate the effectiveness of our proposed framework.

show abstract

Rethinking Collaborative Metric Learning: Toward an Efficient Alternative Without Negative Sampling

Bao

Yang

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

The recently proposed Collaborative Metric Learning (CML) paradigm has aroused wide interest in the area of recommendation systems (RS) owing to its simplicity and effectiveness. Typically, the existing literature of CML depends largely on the negative sampling strategy to alleviate the time-consuming burden of pairwise computation. However, in this work, by taking a theoretical analysis, we find that negative sampling would lead to a biased estimation of the generalization error. Specifically, we show that the samplingbased CML would introduce a bias term in the generalization bound, which is quantified by the per-user Total Variance (TV) between the distribution induced by negative sampling and the ground truth distribution. This suggests that optimizing the sampling-based CML loss function does not ensure a small generalization error even with sufficiently large training data. Moreover, we show that the bias term will vanish without the negative sampling strategy. Motivated by this, we propose an efficient alternative without negative sampling for CML named Sampling-Free Collaborative Metric Learning (SFCML), to get rid of the sampling bias in a practical sense. Finally, comprehensive experiments over seven benchmark datasets speak to the superiority of the proposed algorithm.

show abstract

Stochastic AUC optimization with general loss

Cited by 7 publications

References 17 publications

AUC Maximization in the Era of Big Data and AI: A Survey

AUC Maximization in the Era of Big Data and AI: A Survey

Learning With Multiclass AUC: Theory and Algorithms

Rethinking Collaborative Metric Learning: Toward an Efficient Alternative Without Negative Sampling

Contact Info

Product

Resources

About