Stopping rules and their effect on expected search length

Kraft, Donald H.; Lee, T.

doi:10.1016/0306-4573(79)90007-4

Cited by 43 publications

(42 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings show that stopping strategies based upon the frustration point rule [9] and disgust rule [19], where searchers stop after examining a given number of nonrelevant documents, delivers the best overall performance during simulations. These stopping strategies also best approximated when actual searchers stopped.…”

Section: Introductionmentioning

confidence: 84%

“…Additionally, system-sided research focuses on encoding stopping models into measures and models to evaluate IR system performance [9,15,24], and/or conduct simulations of user interaction [1,4]. However, when people stop is considered a fundamental aspect of human behaviour [26], and as such has been considered more broadly from cognitive and decision-making perspectives [7,19]. Determining when to stop is an inherently di cult task.…”

Section: Related Workmentioning

confidence: 99%

“…Examples include a searcher's prior knowledge of the domain being searched [33], their knowledge and search experience, how they interpret the cues/snippets presented on the Search Engine Results Page (SERP) [35], their understanding of the given information need, and their feeling of enough. To quantify this feeling of "good enough", various researchers have proposed an array of stopping rules and heuristics that try to encode this intuition [6,9,19,26]. These proposals are detailed in Section 2.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Searching and Stopping

Maxwell

Azzopardi

Järvelin

et al. 2015

Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Searching naturally involves stopping points, both at a query level (how far down the ranked list should I go?) and at a session level (how many queries should I issue?). Understanding when searchers stop has been of much interest to the community because it is fundamental to how we evaluate search behaviour and performance. Research has shown that searchers find it di cult to formalise stopping criteria, and typically resort to their intuition of what is "good enough". While various heuristics and stopping criteria have been proposed, little work has investigated how well they perform, and whether searchers actually conform to any of these rules. In this paper, we undertake the first large scale study of stopping rules, investigating how they influence overall session performance, and which rules best match actual stopping behaviour. Our work is focused on stopping at the query level in the context of ad-hoc topic retrieval, where searchers undertake search tasks within a fixed time period. We show that stopping strategies based upon the disgust or frustration point rules -both of which capture a searcher's tolerance to non-relevance -typically result in (i) the best overall performance, and (ii) provide the closest approximation to actual searcher behaviour, although a fixed depth approach also performs remarkably well. Findings from this study have implications regarding how we build measures, and how we conduct simulations of search behaviours.

show abstract

Section: Introductionmentioning

confidence: 84%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Searching and Stopping

Maxwell

Azzopardi

Järvelin

et al. 2015

Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

View full text Add to dashboard Cite

show abstract

“…P @k). The adaptive strategies SS2 and SS3 that consider a user's tolerance to non-relevance are based upon the frustration point and disgust rules, proposed by Cooper [6] and Kraft and Lee [15] respectively. These strategies were selected as they had previously been shown to provide good approximations of actual searcher behaviours [20].…”

Section: Snippet Level Stopping Strategiesmentioning

confidence: 99%

“…As such, researchers have attempted to quantify the sense of "enough" at both levels by proposing a series of stopping rules and heuristics that attempt to encode this intuition (e.g. [3,6,15,25]). Models of stopping behaviour have also been encoded with measures used to evaluate the quality of ranked lists, and within simulations of interaction.…”

Section: Introductionmentioning

confidence: 99%

Information Scent, Searching and Stopping

Maxwell

Azzopardi

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Current models and measures of the Interactive Information Retrieval (IIR) process typically assume that a searcher will always examine the first snippet in a given Search Engine Results Page (SERP), and then with some probability or cutoff, he or she will stop examining snippets and/or documents in the ranked list (snippet level stopping). Prior work has however shown that searchers will form an initial impression of the SERP, and will often abandon a page without clicking on or inspecting in detail any snippets or documents. That is, the information scent affects their decision to continue. In this work, we examine whether considering the information scent of a page leads to better predictions of stopping behaviour. In a simulated analysis, grounded with data from a prior user study, we show that introducing a SERP level stopping strategy can improve the performance attained by simulated users, resulting in an increase in gain across most snippet level stopping strategies. When compared to actual search and stopping behaviour, incorporating SERP level stopping offers a closer approximation than without. These findings show that models and measures that naïvely assume snippets and documents in a ranked list are actually examined in detail are less accurate, and that modelling SERP level stopping is required to create more realistic models of the search process.

show abstract

Evaluating retrieval performance given database and query characteristics: Analytic determination of performance surfaces

Losee

1996

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

An analytic method of information retrieval and filtering evaluation can quantitatively predict the expected number of documents examined in retrieving a relevant document. It also allows researchers and practitioners to qualitatively understand how varying different estimates of query parameter values affects retrieval performance. The incorporation of relevance feedback to increase our knowledge about the parameters of relevant documents and the robustness of parameter estimates is modeled. Single term and two term independence models, as well as a complete term dependence model, are developed. An economic model of retrieval performance may be used to study the effects of database size and to provide analytic answers to questions comparing retrieval from small and large databases, as well as questions about the number of terms in a query. Results are presented as a performance surface, a three dimensional graph showing the effects of two independent variables on performance.

show abstract

Stopping rules and their effect on expected search length

Cited by 43 publications

References 2 publications

Searching and Stopping

Searching and Stopping

Information Scent, Searching and Stopping

Evaluating retrieval performance given database and query characteristics: Analytic determination of performance surfaces

Contact Info

Product

Resources

About