Correcting for Selection Bias in Learning-to-rank Systems

When Inverse Propensity Scoring does not Work

“…A lot of related work has considered the estimation of the position bias parameters θ , using randomization [3,8,13,21], or by jointly estimating relevance and position bias [4,20]. Recently, both Ovaisi et al [16] and Oosterhuis and de Rijke [15] have proposed using different propensities when not all items can be displayed at once (i.e., in case ∃k θ k = 0). For this paper, we will assume that all propensities are positive and thus∆ IPS is unbiased.…”

“…Thus, none of the existing IPS estimators can correct for trust bias or can be adapted to do so. For clarity, this includes: the original CLTR estimators [13,20]; the dual learning algorithm by Ai et al [4]; the IPS with corrections for item-selection bias by Ovaisi et al [16]; the policy aware estimator [15]; and the Bayes-IPS estimator [2].…”

“…As a consequence, an item may receive more clicks because it was displayed at a high rank, not because it was preferred by the user. Other types of bias include item-selection bias: not all items can be displayed at once [15,16]; presentation bias: items are presented in different manners [23]; and trust-bias: users are more likely to click incorrectly on higher ranked items [12]. In order to infer a user's true preferences from their interactions, the effects of these biases have to be corrected for.…”

“…This reweighting compensates for the effect of position bias, allowing CLTR methods to estimate and learn without being affected by position bias in expectation. Later CLTR work has focused on estimating examination probabilities [3,4,8,21], training deep learning models [1], and correcting for more types of bias [2,15,16]. In particular, Agarwal et al [2] have proposed an expansion to IPS to correct for both position bias and trust bias.…”

Unbiased Learning to Rank: Counterfactual and Online Approaches

“…A prominent form of bias in ranking is position bias: users devote more attention to higher ranked documents, and consequently, the order in which documents are displayed affects the interactions that take place [6]. Another common form of bias is item selection bias: users can only interact with documents that are displayed; hence, the selection of displayed documents heavily affects which interactions are possible [18]. Naively ignoring these biases during the learning process will result in biased ranking models that are not fully optimized for user preferences [11].…”