Cross-Positional Attention for Debiasing Clicks

Zhuang, Honglei; Qin, Zhen; Wang, Xuanhui; Bendersky, Michael; Qian, Xinyu; Hu, Po; Chen, Dan Chary

doi:10.1145/3442381.3450098

Cited by 22 publications

(15 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because both the likelihood and the solution found by Newton's method appear intractable, we are unable to prove this estimator is unbiased. Nonetheless, maximum likelihood estimators have been very effective when applied to user interactions in the past [6,68] and can be consistent under certain feasible conditions [9].…”

Section: Bandwagon Effect Mitigation 51 Correcting For the Bandwagon ...mentioning

confidence: 99%

The Bandwagon Effect: Not Just Another Bias

Knyazev,

Oosterhuis

2022

Preprint

View full text Add to dashboard Cite

Optimizing recommender systems based on user interaction data is mainly seen as a problem of dealing with selection bias, where most existing work assumes that interactions from different users are independent. However, it has been shown that in reality user feedback is often influenced by earlier interactions of other users, e.g. via average ratings, number of views or sales per item, etc. This phenomenon is known as the bandwagon effect.In contrast with previous literature, we argue that the bandwagon effect should not be seen as a problem of statistical bias. In fact, we prove that this effect leaves both individual interactions and their sample mean unbiased. Nevertheless, we show that it can make estimators inconsistent, introducing a distinct set of problems for convergence in relevance estimation. Our theoretical analysis investigates the conditions under which the bandwagon effect poses a consistency problem and explores several approaches for mitigating these issues. This work aims to show that the bandwagon effect poses an underinvestigated open problem that is fundamentally distinct from the well-studied selection bias in recommendation. CCS CONCEPTS• Information systems → Social recommendation; Learning to rank; • Mathematics of computing → Time series analysis.

show abstract

Section: Bandwagon Effect Mitigation 51 Correcting For the Bandwagon ...mentioning

confidence: 99%

The Bandwagon Effect: Not Just Another Bias

Knyazev,

Oosterhuis

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With the inception of unbiased LTR, click models became a common choice for estimating bias parameters for counterfactual estimation instead of relevance estimation [45,66,68,69]. However, recently novel click models for relevance estimation have been introduced again: For example, Vardasbi et al [65] proposed a click model that uses a prior distribution over relevances; and Zhuang et al [75] used a click model with a state-of-the-art deep learning architecture for relevance estimation in a grid layout. They found their click model to be more effective than existing counterfactual estimation techniques in a real-world recommendation setting.…”

Section: Click Modellingmentioning

confidence: 99%

“…Thus far, this approach has been very successful as methods have been found for an exceeding number of behavior models [33,41,44,66,68]. The large number of recent publications make it evident that the success of this approach is still ongoing and that unbiased LTR remains a very active field [1,2,5,6,27,41,44,46,48,65,66,75].…”

Section: Introductionmentioning

confidence: 99%

“…Our method of finding the limitations of the prevalent approach is to invert it: we start with generic formulations that describe the main branches of existing methods in the unbiased LTR field: counterfactual estimation [33,41,68] and click modelling [15,65,75]. Subsequently, from these generic descriptions, we derive the conditions under which they can be unbiased.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank

Oosterhuis

2022

Preprint

View full text Add to dashboard Cite

Click-based learning to rank (LTR) tackles the mismatch between click frequencies on items and their actual relevance. The approach of previous work has been to assume a model of click behavior and to subsequently introduce a method for unbiasedly estimating preferences under that assumed model. The success of this approach is evident in that unbiased methods have been found for an increasing number of behavior models and types of bias.This work aims to uncover the implicit limitations of the highlevel prevalent approach in the counterfactual LTR field. Thus, in contrast with limitations that follow from explicit assumptions, our aim is to recognize limitations that the field is currently unaware of. We do this by inverting the existing approach: we start by capturing existing methods in generic terms, and subsequently, from these generic descriptions we derive the click behavior for which these methods can be unbiased. Our inverted approach reveals that there are indeed implicit limitations to the counterfactual LTR approach: we find counterfactual estimation can only produce unbiased methods for click behavior based on affine transformations. In addition, we also recognize previously undiscussed limitations of click-modelling and pairwise approaches to click-based LTR. Our findings reveal that it is impossible for existing approaches to provide unbiasedness guarantees for all plausible click behavior models. CCS CONCEPTS• Information systems → Learning to rank.

show abstract

“…In order to efficiently make use of user interaction data in learning of ranking models, studies on alleviating biases in user interaction data have been conducted, called Unbiased Learning to Rank (ULTR) [4,5,18] or Counterfactual Learning to Rank (CLTR) [1,19,29]. Previously, studies focused on position bias and usually assumed that users can examine the whole ranking list so that every relevant document is guaranteed to be examined [7,12,25,50]. Due to the limitation of the device sizes, however, search engines usually only display at most 𝑘 relevant documents to the user-issued query, on the basis of existing ranking models.…”

Section: Introductionmentioning

confidence: 99%

Unbiased Top-k Learning to Rank with Causal Likelihood Decomposition

Zhao¹,

Xu²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unbiased learning to rank has been proposed to alleviate the biases in the search ranking, making it possible to train ranking models with user interaction data. In real applications, search engines are designed to display only the most relevant 𝑘 documents from the retrieved candidate set. The rest candidates are discarded. As a consequence, position bias and sample selection bias usually occur simultaneously. Existing unbiased learning to rank approaches either focus on one type of bias (e.g., position bias) or mitigate the position bias and sample selection bias with separate components, overlooking their associations. In this study, we first analyze the mechanisms and associations of position bias and sample selection bias from the viewpoint of a causal graph. Based on the analysis, we propose Causal Likelihood Decomposition (CLD), a unified approach to simultaneously mitigating these two biases in top-𝑘 learning to rank. By decomposing the log-likelihood of the biased data as an unbiased term that only related to relevance, plus other terms related to biases, CLD successfully detaches the relevance from position bias and sample selection bias. An unbiased ranking model can be obtained from the unbiased term, via maximizing the whole likelihood. An extension to the pairwise neural ranking is also developed. Advantages of CLD include theoretical soundness and a unified framework for pointwise and pairwise unbiased top-𝑘 learning to rank. Extensive experimental results verified that CLD, including its pairwise neural extension, outperformed the baselines by mitigating both the position bias and the sample selection bias. Empirical studies also showed that CLD is robust to the variation of bias severity and the click noise. CCS CONCEPTS• Information systems → Learning to rank; • Computing methodologies → Learning from implicit feedback.

show abstract

Cross-Positional Attention for Debiasing Clicks

Cited by 22 publications

References 41 publications

The Bandwagon Effect: Not Just Another Bias

The Bandwagon Effect: Not Just Another Bias

Reaching the End of Unbiasedness: Uncovering Implicit Limitations of Click-Based Learning to Rank

Unbiased Top-k Learning to Rank with Causal Likelihood Decomposition

Contact Info

Product

Resources

About