Addressing Trust Bias for Unbiased Learning-to-Rank

Agarwal, Aman; Wang, Xuanhui; Li, Cheng; Bendersky, Michael; Najork, Marc

doi:10.1145/3308558.3313697

Cited by 58 publications

(121 citation statements)

References 24 publications

Supporting

Mentioning

115

Contrasting

Order By: Relevance

“…In this section, we discuss Agarwal et al [2]'s Bayes-IPS method designed specifically for trust bias. We prove that no IPS estimator is able to correct for trust bias, including Bayes-IPS.…”

Section: Existing Methods and Trust Biasmentioning

confidence: 99%

“…Recently, Agarwal et al [2] have modeled trust bias by distinguishing between perceived relevanceR ∈ {0, 1} and real relevance R. Trust bias occurs because users are more likely to perceive items as relevantR = 1 if they are among the top ranked items in the list. In Agarwal et al's model, a click happens when a user examines and perceives an item to be relevant: C = 1 ↔ E = 1 ∧R = 1.…”

Section: Trust Biasmentioning

confidence: 99%

“…We will use this property in Section 5 to introduce affine corrections for these biases. An empirical analysis by Agarwal et al [2] shows that their trust bias model better captures observed user behavior than the model that only considers position bias (Eq. 4).…”

Section: Trust Biasmentioning

confidence: 99%

“…This reweighting compensates for the effect of position bias, allowing CLTR methods to estimate and learn without being affected by position bias in expectation. Later CLTR work has focused on estimating examination probabilities [3,4,8,21], training deep learning models [1], and correcting for more types of bias [2,15,16]. In particular, Agarwal et al [2] have proposed an expansion to IPS to correct for both position bias and trust bias.…”

Section: Introductionmentioning

confidence: 99%

“…We identify the root cause to be the fact that IPS only corrects for Missing-Not-At-Random (MNAR) feedback [13]. While position bias prevents clicks from occurring due to a lack of user examination, trust bias adds additional clicks due to user trust [2,12]. an item (to be ranked) D q set of items to be ranked for query q λ metric function that assigns a weight per rank f a ranker, or ranking function, that scores items y i a ranking displayed at interaction i C a click on an item E user examination of an item R the relevance of an item R the perceived relevance of an item θ k examination probability at rank k:…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

When Inverse Propensity Scoring does not Work

Vardasbi

Oosterhuis

Rijke

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Besides position bias, which has been well-studied, trust bias is another type of bias prevalent in user interactions with rankings: users are more likely to click incorrectly w.r.t. their preferences on highly ranked items because they trust the ranking system. While previous work has observed this behavior in users, we prove that existing Counterfactual Learning to Rank (CLTR) methods do not remove this bias, including methods specifically designed to mitigate this type of bias. Moreover, we prove that Inverse Propensity Scoring (IPS) is principally unable to correct for trust bias under non-trivial circumstances. Our main contribution is a new estimator based on affine corrections: it both reweights clicks and penalizes items displayed on ranks with high trust bias. Our estimator is the first estimator that is proven to remove the effect of both trust bias and position bias. Furthermore, we show that our estimator is a generalization of the existing CLTR framework: if no trust bias is present, it reduces to the original IPS estimator. Our semi-synthetic experiments indicate that by removing the effect of trust bias in addition to position bias, CLTR can approximate the optimal ranking system even closer than previously possible.

show abstract

Section: Existing Methods and Trust Biasmentioning

confidence: 99%

Section: Trust Biasmentioning

confidence: 99%

Section: Trust Biasmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

When Inverse Propensity Scoring does not Work

Vardasbi

Oosterhuis

Rijke

2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

show abstract

The Impact of User Demographics and Task Types on Cross-App Mobile Search

Aliannejadi

Crestani

Huibers

et al. 2021

Flexible Query Answering Systems

View full text Add to dashboard Cite

User interaction data is an important source of supervision in counterfactual learning to rank (CLTR). Such data suffers from presentation bias. Much work in unbiased learning to rank (ULTR) focuses on position bias, i.e., items at higher ranks are more likely to be examined and clicked. Inter-item dependencies also influence examination probabilities, with outlier items in a ranking as an important example. Outliers are defined as items that observably deviate from the rest and therefore stand out in the ranking. In this paper, we identify and introduce the bias brought about by outlier items: users tend to click more on outlier items and their close neighbors.To this end, we first conduct a controlled experiment to study the effect of outliers on user clicks. Next, to examine whether the findings from our controlled experiment generalize to naturalistic situations, we explore real-world click logs from an e-commerce platform. We show that, in both scenarios, users tend to click significantly more on outlier items than on non-outlier items in the same rankings. We show that this tendency holds for all positions, i.e., for any specific position, an item receives more interactions when presented as an outlier as opposed to a non-outlier item. We conclude from our analysis that the effect of outliers on clicks is a type of bias that should be addressed in ULTR. We therefore propose an outlieraware click model that accounts for both outlier and position bias, called outlier-aware position-based model (OPBM). We estimate click propensities based on OPBM; through extensive experiments performed on both real-world e-commerce data and semi-synthetic data, we verify the effectiveness of our outlier-aware click model. Our results show the superiority of OPBM against baselines in terms of ranking performance and true relevance estimation. CCS CONCEPTS• Information systems → Learning to rank.

show abstract

Implications and New Directions for IR Research and Practices

Liu

2023

The Information Retrieval Series

View full text Add to dashboard Cite

Addressing Trust Bias for Unbiased Learning-to-Rank

Cited by 58 publications

References 24 publications

When Inverse Propensity Scoring does not Work

When Inverse Propensity Scoring does not Work

The Impact of User Demographics and Task Types on Cross-App Mobile Search

Implications and New Directions for IR Research and Practices

Contact Info

Product

Resources

About