Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm

Abstract

Recently a number of algorithms under the theme of 'unbiased learning-to-rank' have been proposed, which can reduce position bias, the major type of bias in click data, and train a highperformance ranker with click data in learning-to-rank. Most of the existing algorithms, based on the inverse propensity weighting (IPW) principle, first estimate the click bias at each position, and then train an unbiased ranker with the estimated biases using a learning-to-rank algorithm. However, there has not been a method for unbiased pairwise learning-to-rank that can simultaneously conduct debiasing of click data and training of a ranker using a pairwise loss function. In this paper, we propose a novel framework to accomplish the goal and apply this framework to the state-ofthe-art pairwise learning-to-rank algorithm, LambdaMART. Our algorithm named Unbiased LambdaMART can jointly estimate the biases at click positions and the biases at unclick positions, and learn an unbiased ranker. Experiments on benchmark data show that Unbiased LambdaMART can significantly outperform existing algorithms by large margins. In addition, an online A/B Testing at a commercial search engine shows that Unbiased LambdaMART can effectively conduct debiasing of click data and enhance relevance ranking.

When Inverse Propensity Scoring does not Work

“…Importantly, we have to limit the possible choices for ρ, because trivially unbiased estimators are theoretically possible [9]:…”

Bridging the Gap between Click and Relevance for Learning-to-Rank with Minimal Supervision

“…That is, the main challenge is how to bridge the gap between click and relevance. For that, existing work [5] introduces two assumptions: (i) the clicked and the unclicked probability of documents are proportional to the relevance and the irrelevance probability, respectively. (ii) The bias of clicked and unclicked documents are independent of each other.…”

“…Also, Wang et al [13] developed a regression-based EM model to estimate the position bias of click data, and Ai et al [1] utilized a dual learning model that jointly estimates position bias and trains an unbiased ranker. Most recently, Hu et al [5] developed a pairwise unbiased ranker in LambdaMART [3].…”

“…Although existing studies [1,2,5,9,12,13] have shown that eliminating biases from click data can improve the performance of the ranker, they pose two key challenges: (i) current studies solely focus on eliminating a specific bias type, e.g., position bias, even though various biases exist in click data. When considering multiple bias types, it is inevitable to combine multiple propensity weights, which is too complicated and may need high computational cost.…”

“…After learning the confidence estimator, we incorporate the confidence weights into the learning-to-rank model. Inspired by unbiased LambdaMART [5], we combine the confidence weight with LambdaMART [3], namely wLambdaMART. That is, we infer confidence weights for all click data, and apply them to LambdaMART.…”