Extreme Regression for Dynamic Search Advertising

Abstract

This paper introduces a new learning paradigm called eXtreme Regression (XR) whose objective is to accurately predict the numerical degrees of relevance of an extremely large number of labels to a data point. XR can provide elegant solutions to many large-scale ranking and recommendation applications including Dynamic Search Advertising (DSA). XR can learn more accurate models than the recently popular extreme classifiers which incorrectly assume strictly binary-valued label relevances. Traditional regression metrics which sum the errors over all the labels are unsuitable for XR problems since they could give extremely loose bounds for the label ranking quality. Also, the existing regression algorithms won't efficiently scale to millions of labels. This paper addresses these limitations through: (1) new evaluation metrics for XR which sum only the k largest regression errors; (2) a new algorithm called XReg which decomposes XR task into a hierarchy of much smaller regression problems thus leading to highly efficient training and prediction. This paper also introduces a (3) new labelwise prediction algorithm in XReg useful for DSA and other recommendation tasks.Experiments on benchmark datasets demonstrated that XReg can outperform the state-of-the-art extreme classifiers as well as large-scale regressors and rankers by up to 50% reduction in the new XR error metric, and up to 2% and 2.4% improvements in terms of the propensity-scored precision metric used in extreme classification and the click-through rate metric used in DSA respectively. Deployment of XReg on DSA in Bing resulted in a relative gain of 27% in query coverage. XReg's source code can be downloaded from [1]

BoostLR: A Boosting-Based Learning Ensemble for Label Ranking Tasks

“…In meta-learning domains, the characteristics of each problem instance are considered, and the output is an ordered list of algorithms according to their suitability to the given problem [6], [7]. Lastly, In text classification, a label ranking algorithm can be employed to output a ranked list of topics, tags or advertisements for a document or web page (the instance) [8], [9]. Due to this wide applicability, label ranking has recently attracted a lot of focus from the machine learning community [10]- [20].…”

Generalized Zero-Shot Extreme Multi-label Learning

“…(1) State of the art extreme classifiers such as AttentionXML [66], Astec [11], DiSMEC [2], Parabel [45] and Bonsai [26] (2) Extreme classifiers which improve performance on few-shot labels such as DECAF [40], XReg [46] and PFastreXML [20] (3) Dense retrieval methods based on the state of the art natural language modelling architectures such as Sentence BERT bi-encoder [48], Fasttext [24] and WarpLDA (topic model) [10], these algorithms provide strong scalable baseline to compare ZestXML's performance over zero-shot and few-shot labels (4) Leading zero-shot multi-label learners such as 0-BIGRU-WLAN, 0-CNN-LWAN [50] and CoNSE [43], these baselines don't scale on extreme datasets, hence, ZestXML's comparison against these baselines is reported only for EURLex-4.3K in Table ??. The implementation of all the aforementioned algorithms were provided by their authors.…”

“…Additionally, they also tend to perform poorly on few-shot labels due to classifier over-fitting issues (see Section 5). Recently, several approaches have been proposed which aim to model the few-shot labels more accurately [40,46]; these, however, do not address the zero-shot prediction problem.…”

“…AttentionXML achieves state-of-the-art prediction accuracies through BiLSTM-based feature learning but it is slow to train even with high-end GPUs and does not scale to more than a few million labels. Extreme classifiers like PfastreXML [20], XReg [46] and DECAF [40] optimize for few-shot labels where the first two learn for propensity-scored losses which prioritize tail labels and the last incorporates label features for better tail label modeling. Regardless of their nature, none of these perform zeroshot label prediction.…”

Extreme Multi-label Learning for Semantic Matching in Product Search

“…For instance, the deep learning XMC model X-Transformer [7,52] achieves state-of-the-art performance on public academic benchmarks [3]. Partition-based methods such as Parabel [33] and XReg [34], as another example, finds successful applications to dynamic search advertising in Bing. In particular, tree-based partitioning XMC models are a staple of modern search engines and recommender systems due to their inference time being sub-linear (i.e., logarithmic) to the enormous output space (e.g., 100 million or more).…”