Investigating the Reliability of Click Models

Mao, Jiaxin; Chu, Zhumin; Liu, Yiqun; Zhang, Min; Ma, Shaoping

doi:10.1145/3341981.3344242

Cited by 8 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The quality of click models is often evaluated by the Log-Likelihood and Perplexity [61], but also other reliability measures exist [62]. In previous work, click models have mainly been evaluated on semi-public web search datasets, e.g., from Yahoo!…”

Section: Click Modelsmentioning

confidence: 99%

Validating Synthetic Usage Data in Living Lab Environments

Breuer,

Fuhr,

Schaer

2024

J. Data and Information Quality

View full text Add to dashboard Cite

Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for small-scale platforms to validate information retrieval systems with real users. If enough user interaction data is available, click models can be parameterized from historical sessions to evaluate systems before exposing users to experimental rankings. However, interaction data is sparse in living labs, and little is studied about how click models can be validated for reliable user simulations when click data is available in moderate amounts. This work introduces an evaluation approach for validating synthetic usage data generated by click models in data-sparse human-in-the-loop environments like living labs. We ground our methodology on the click model’s estimates about a system ranking compared to a reference ranking for which the relative performance is known. Our experiments compare different click models and their reliability and robustness as more session log data becomes available. In our setup, simple click models can reliably determine the relative system performance with already 20 logged sessions for 50 queries. In contrast, more complex click models require more session data for reliable estimates, but they are a better choice in simulated interleaving experiments when enough session data is available. While it is easier for click models to distinguish between more diverse systems, it is harder to reproduce the system ranking based on the same retrieval algorithm with different interpolation weights. Our setup is entirely open, and we share the code to reproduce the experiments.

show abstract

Section: Click Modelsmentioning

confidence: 99%

Validating Synthetic Usage Data in Living Lab Environments

Breuer,

Fuhr,

Schaer

2024

J. Data and Information Quality

View full text Add to dashboard Cite

show abstract

“…Wang et al [48] unify the training of the ranker model and the estimation of examination propensity with a graphical model and an EM algorithm. Despite their success, one major drawback of click models is that they usually require that the same query-document pair appears multiple times for reliable inference [30]; thus they may fall short for tail queries. The other school derives from counterfactual learning, which treats bias as a counterfactual factor and debiases user clicks via inverse propensity weighting [25,47].…”

Section: Related Workmentioning

confidence: 99%

Model-based Unbiased Learning to Rank

Luo¹,

Zou²,

Ai³

et al. 2022

Preprint

View full text Add to dashboard Cite

Unbiased Learning to Rank (ULTR) that learns to rank documents with biased user feedback data is a well-known challenge in information retrieval. Existing methods in unbiased learning to rank typically rely on click modeling or inverse propensity weighting (IPW). Unfortunately, the search engines are faced with severe long-tail query distribution, where neither click modeling nor IPW can handle well. Click modeling suffers from data sparsity problem since the same query-document pair appears limited times on tail queries; IPW suffers from high variance problem since it is highly sensitive to small propensity score values. Therefore, a general debiasing framework that works well under tail queries is in desperate need. To address this problem, we propose a model-based unbiased learning-to-rank framework. Specifically, we develop a general context-aware user simulator to generate pseudo clicks for unobserved ranked lists to train rankers, which addresses the data sparsity problem. In addition, considering the discrepancy between pseudo clicks and actual clicks, we take the observation of a ranked list as the treatment variable and further incorporate inverse propensity weighting with pseudo labels in a doubly robust way. The derived bias and variance indicate that the proposed model-based method is more robust than existing methods. Finally, extensive experiments on benchmark datasets, including simulated datasets and real click logs, demonstrate that the proposed model-based method consistently performs outperforms state-of-the-art methods in various scenarios.

show abstract

“…Authenticity is the estimation of system error (i.e., estimation deviation), while accuracy is the estimation of random error (i.e., estimation variance). Mao et al (2019) suggested to study the reliability of correlation estimates derived from click models. The posteriori distribution of correlation parameters is inferred by the method of variable decibels instead of the point estimation of correlation.…”

Section: Measuring User Experience From Behavior Datamentioning

confidence: 99%

Measuring and Improving User Experience Through Artificial Intelligence-Aided Design

Yang

Wei

2020

Front. Psychol.

View full text Add to dashboard Cite

This paper aims to propose a methodology for measuring user experience (UX) by using artificial intelligence-aided design (AIAD) technology in mobile application design. Unlike the traditional assistance design tools, AIAD focuses on the rational use of artificial intelligence (AI) technology to measure and improve UX since conventional data collection methods (such as user interview and user observation) for user behavior data are inefficient and time-consuming. We propose to obtain user behavior data from logs of mobile application. In order to protect the privacy of users, only a few dimensions of information is used in the process of browsing and operating mobile application. The goal of the proposed methodology is to make the deep neural network model simulate the user's experience in the process of operating a mobile application as much as possible. We design and use projected pages of application to train neural networks for specific tasks. These projected pages consist of the click information of all users in the process of completing a certain task. Thus, features of user behavior can be aggregated and mapped in the connection layers and the hidden layers. Finally, the optimized design is executed on the social communication application to verify the efficiency of the proposed methodology.

show abstract

Investigating the Reliability of Click Models

Cited by 8 publications

References 11 publications

Validating Synthetic Usage Data in Living Lab Environments

Validating Synthetic Usage Data in Living Lab Environments

Model-based Unbiased Learning to Rank

Measuring and Improving User Experience Through Artificial Intelligence-Aided Design

Contact Info

Product

Resources

About