Constructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model

Chen, Nuo; Zhang, Fan; Sakai, Tetsuya

doi:10.1145/3477495.3531953

Cited by 12 publications

(5 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They adopted users' annotations to represent the perceptions and found that the real-time experience affected users' perceptions. These effects can be explained by theories of anchoring bias, reference-dependence bias, and expectation confirmation, which support that people usually refer to a reference point when evaluating current situation (Chen et al, 2022;Hossain & Quaddus, 2012;Liu & Han, 2020;Shokouhi et al, 2015;Thomas et al, 2022). The reference-dependence effect is a key component of prospect theory, explaining the cognitive bias that occurs when users evaluate current outcomes based on a previous reference point (Liu & Han, 2020;Tversky & Kahneman, 1992).…”

Section: Introductionmentioning

confidence: 89%

Understanding users' dynamic perceptions of search gain and cost in sessions: An expectation confirmation model

Wang,

Liu

2024

Asso for Info Science & Tech

View full text Add to dashboard Cite

Understanding the roles of search gain and cost in users' search decision‐making is a key topic in interactive information retrieval (IIR). While previous research has developed user models based on simulated gains and costs, it is unclear how users' actual perceptions of search gains and costs form and change during search interactions. To address this gap, our study adopted expectation‐confirmation theory (ECT) to investigate users' perceptions of gains and costs. We re‐analyzed data from our previous study, examining how contextual and search features affect users' perceptions and how their expectation‐confirmation states impact their following searches. Our findings include: (1) The point where users' actual dwell time meets their constant expectation may serve as a reference point in evaluating perceived gain and cost; (2) these perceptions are associated with in situ experience represented by usefulness labels, browsing behaviors, and queries; (3) users' current confirmation states affect their perceptions of Web page usefulness in the subsequent query. Our findings demonstrate possible effects of expectation‐confirmation, prospect theory, and information foraging theory, highlighting the complex relationships among gain/cost, expectations, and dwell time at the query level, and the reference‐dependent expectation at the session level. These insights enrich user modeling and evaluation in human‐centered IR.

show abstract

Section: Introductionmentioning

confidence: 89%

Understanding users' dynamic perceptions of search gain and cost in sessions: An expectation confirmation model

Wang,

Liu

2024

Asso for Info Science & Tech

View full text Add to dashboard Cite

show abstract

“…Thus, their expectations create cognitive bias and lead to biased results in user modeling and evaluation. Previous studies have been engaged in modeling users' behaviors under influence of their expectations (N. Chen, Zhang, & Sakai, 2022; Moffat, Bailey, Scholer, & Thomas, 2017). However, there is little direct evidence showing relationships between users' expectations and actual search behaviors.…”

Section: Introductionmentioning

confidence: 99%

Investigating the Relationship between In‐Situ User Expectations and Web Search Behavior

Wang

Liu

2022

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

Pre‐adoption expectations often serve as an implicit reference point in users' evaluation of information systems and are closely associated with their goals of interactions, behaviors, and overall satisfaction. Previous studies have involved simulated user expectation as a feature in user modeling to model biased search actions. However, there is still little direct evidence revealing the relationships between users' expectations and their actual search behaviors. To address the gap, we collected 448 query sessions from participants in a controlled‐lab user expectation study and gathered direct query‐level feedback on their expected information gains (e.g., number of useful pages) and expected search efforts (e.g., clicks and dwell time) under each query. To our knowledge, this is the first attempt to explicitly examine the connections between different aspects of in‐situ search expectations and user behaviors. Findings on user expectation advance our understanding of users' search decision‐making and evaluation strategies and will also facilitate the design and evaluation of expectation‐aware user models, metrics, and IR systems.

show abstract

“…As described in Section 2.1, most offline evaluation metrics treat users as globally rational decision makers when simulating interactions with search engines, but this assumption has been increasingly challenged recently. In recent years, with the growing knowledge about users' cognitive biases, some works in the field of IR system evaluation began to introduce cognitive biases into the construction and meta-evaluation of evaluation metrics [14,17,107]. However, there is few work that incorporates the decoy effect into the calculation of IR system evaluation metrics.…”

Section: The Evaluation Of Information Retrieval Systemsmentioning

confidence: 99%

“…Therefore, there is a need to develop new evaluation metrics to more accurately measure system performance and ensure they reflect users' actual experiences and preferences. Although some recent efforts have incorporated cognitive biases into the calculation of IR system evaluation metrics [14,17,107], they have not considered the decoy effect. The introduction of DEJA-VU expands the work on IR system evaluation by considering the vulerability of IR systems to the decoy effect.…”

Section: Main Findings and Implicationsmentioning

confidence: 99%

See 1 more Smart Citation

A study of the motivation of clerical officers in the Hong Kong Polytechnic University

張元妹¹

View full text Add to dashboard Cite

Automatic fact-checking plays a crucial role in combating the spread of misinformation. Large Language Models (LLMs) and Instruction-Following variants, such as InstructGPT and Alpaca, have shown remarkable performance in various natural language processing tasks. However, their knowledge may not always be up-to-date or sufficient, potentially leading to inaccuracies in fact-checking. To address this limitation, we propose combining the power of instruction-following language models with external evidence retrieval to enhance fact-checking performance. Our approach involves leveraging search engines to retrieve relevant evidence for a given input claim. This external evidence serves as valuable supplementary information to augment the knowledge of the pretrained language model. Then, we instruct-tune an open-sourced language model, called LLaMA, using this evidence, enabling it to predict the veracity of the input claim more accurately. To evaluate our method, we conducted experiments on two widely used fact-checking datasets: RAWFC and LIAR. The results demonstrate that our approach achieves state-of-the-art performance in fact-checking tasks. By integrating external evidence, we bridge the gap between the model's knowledge and the most up-to-date and sufficient context available, leading to improved fact-checking outcomes. Our findings have implications for combating misinformation and promoting the dissemination of accurate information on online platforms. Our released materials are accessible at: https://thcheung.github.io/factllama.

show abstract

Constructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model

Cited by 12 publications

References 34 publications

Understanding users' dynamic perceptions of search gain and cost in sessions: An expectation confirmation model

Understanding users' dynamic perceptions of search gain and cost in sessions: An expectation confirmation model

Investigating the Relationship between In‐Situ User Expectations and Web Search Behavior

A study of the motivation of clerical officers in the Hong Kong Polytechnic University

Contact Info

Product

Resources

About