Model Ensemble for Click Prediction in Bing Search Ads

Ling, Xiaoliang; Deng, Weiwei; Gu, Chen; Zhou, Hucheng; Li, Cui; Sun, Feng

doi:10.1145/3041021.3054192

Cited by 111 publications

(62 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As aforementioned, the study targeted at Ads CTR [27] also considered combining NNs and DTs. However, some opposite conclusions were drawn because of the difference in the settings.…”

Section: Main Evaluation Results and Analysismentioning

confidence: 99%

See 1 more Smart Citation

Combining Decision Trees and Neural Networks for Learning-to-Rank in Personal Search

Qin

Wang

et al. 2019

Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Decision Trees (DTs) like LambdaMART have been one of the most effective types of learning-to-rank algorithms in the past decade. They typically work well with hand-crafted dense features (e.g., BM25 scores). Recently, Neural Networks (NNs) have shown impressive results in leveraging sparse and complex features (e.g., query and document keywords) directly when a large amount of training data is available. While there is a large body of work on how to use NNs for semantic matching between queries and documents, relatively less work has been conducted to compare NNs with DTs for general learning-to-rank tasks, where dense features are also available and DTs can achieve state-of-the-art performance. In this paper, we study how to combine DTs and NNs to effectively bring the benefits from both sides in the learning-to-rank setting. Specifically, we focus our study on personal search where clicks are used as the primary labels with unbiased learning-to-rank algorithms and a significantly large amount of training data is easily available. Our combination methods are based on ensemble learning. We design 12 variants and compare them based on two aspects, ranking effectiveness and ease-of-deployment, using two of the largest personal search services: Gmail search and Google Drive search. We show that direct application of existing ensemble methods can not achieve both aspects. We thus design a novel method that uses NNs to compensate DTs via boosting. We show that such a method is not only easier to deploy, but also gives comparable or better ranking accuracy.

show abstract

“…As aforementioned, the study targeted at Ads CTR [27] also considered combining NNs and DTs. However, some opposite conclusions were drawn because of the difference in the settings.…”

Section: Main Evaluation Results and Analysismentioning

confidence: 99%

“…Another recent study also considered combining NNs and DTs [27] while the target was to improve ads CTR instead of ranking performance in search. Besides the difference in set-ups, the combining strategies based on NNs-boosting-DTs were missed, which are demonstrated to be optimal according to our study.…”

Section: Related Workmentioning

confidence: 99%

Combining Decision Trees and Neural Networks for Learning-to-Rank in Personal Search

Qin

Wang

et al. 2019

Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

show abstract

“…Our system is akin to the one described in [14], which consists of several steps including selection, relevance filtration, CTR prediction, ranking and allocation.…”

Section: Related Workmentioning

confidence: 99%

Learning Fast Matching Models from Weak Annotations

Luo

Sun

et al. 2019

The World Wide Web Conference

View full text Add to dashboard Cite

We propose a novel training scheme for fast matching models in Search Ads, motivated by practical challenges. The first challenge stems from the pursuit of high throughput, which prohibits the deployment of inseparable architectures, and hence greatly limits model accuracy. The second problem arises from the heavy dependency on human provided labels, which are expensive and time-consuming to collect, yet how to leverage unlabeled search log data is rarely studied. The proposed training framework targets on mitigating both issues, by treating the stronger but undeployable models as annotators, and learning a deployable model from both human provided relevance labels and weakly annotated search log data. Specifically, we first construct multiple auxiliary tasks from the enumerated relevance labels, and train the annotators by jointly learning from those related tasks. The annotation models are then used to assign scores to both labeled and unlabeled training samples. The deployable model is firstly learnt on the scored unlabeled data, and then fine-tuned on scored labeled data, by leveraging both labels and scores via minimizing the proposed label-aware weighted loss. According to our experiments, compared with the baseline that directly learns from relevance labels, training by the proposed framework outperforms it by a large margin, and improves data efficiency substantially by dispensing with 80% labeled samples. The proposed framework allows us to improve the fast matching model by learning from stronger annotators while keeping its architecture unchanged. Meanwhile, it offers a principled manner to leverage search log data in the training phase, which could effectively alleviate our dependency on human provided labels.

show abstract

“…In terms of techniques, existing works have also considered combining tree-based and embedding-based models, among which the most popular method is boosting [11,27,49]. These solutions typically perform a late fusion on the prediction of two kinds of models.…”

Section: Related Workmentioning

confidence: 99%

“…Particularly, GB-CENT achieves CF effect by conducting MF over categorical features; meanwhile, it employs GBDT on the supporting instances of numerical features to capture the nonlinear feature interactions. Ling et al [27] shows that boosting neural networks with GBDT achieves the best performance in the CTR prediction. However, these boosting methods only fuse the outputs of different models and may be insufficient to fully propagate information between tree-based and embedding-based models.…”

Section: Related Workmentioning

confidence: 99%

Tem

Wang

Feng

et al. 2018

Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18

161

View full text Add to dashboard Cite

While collaborative filtering is the dominant technique in personalized recommendation, it models user-item interactions only and cannot provide concrete reasons for a recommendation. Meanwhile, the rich side information affiliated with user-item interactions (e.g., user demographics and item attributes), which provide valuable evidence that why a recommendation is suitable for a user, has not been fully explored in providing explanations.On the technical side, embedding-based methods, such as Wide&Deep and neural factorization machines, provide state-ofthe-art recommendation performance. However, they work like a black-box, for which the reasons underlying a prediction cannot be explicitly presented. On the other hand, tree-based methods like decision trees predict by inferring decision rules from data. While being explainable, they cannot generalize to unseen feature interactions thus fail in collaborative filtering applications.In this work, we propose a novel solution named Tree-enhanced Embedding Method that combines the strengths of embeddingbased and tree-based models. We first employ a tree-based model to learn explicit decision rules (aka. cross features) from the rich side information. We next design an embedding model that can incorporate explicit cross features and generalize to unseen cross features on user ID and item ID. At the core of our embedding method is an easy-to-interpret attention network, making the recommendation process fully transparent and explainable. We conduct experiments on two datasets of tourist attraction and restaurant recommendation, demonstrating the superior performance and explainability of our solution. CCS CONCEPTS• Information systems → Recommender systems; KEYWORDS Explainable Recommendation, Tree-based Model, Embedding-based Model, Neural Attention Network * Xiangnan He is the corresponding author. This paper is published under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.

show abstract

Model Ensemble for Click Prediction in Bing Search Ads

Cited by 111 publications

References 19 publications

Combining Decision Trees and Neural Networks for Learning-to-Rank in Personal Search

Combining Decision Trees and Neural Networks for Learning-to-Rank in Personal Search

Learning Fast Matching Models from Weak Annotations

Tem

Contact Info

Product

Resources

About