Searching and Stopping

Abstract

Searching naturally involves stopping points, both at a query level (how far down the ranked list should I go?) and at a session level (how many queries should I issue?). Understanding when searchers stop has been of much interest to the community because it is fundamental to how we evaluate search behaviour and performance. Research has shown that searchers find it di cult to formalise stopping criteria, and typically resort to their intuition of what is "good enough". While various heuristics and stopping criteria have been proposed, little work has investigated how well they perform, and whether searchers actually conform to any of these rules. In this paper, we undertake the first large scale study of stopping rules, investigating how they influence overall session performance, and which rules best match actual stopping behaviour. Our work is focused on stopping at the query level in the context of ad-hoc topic retrieval, where searchers undertake search tasks within a fixed time period. We show that stopping strategies based upon the disgust or frustration point rules -both of which capture a searcher's tolerance to non-relevance -typically result in (i) the best overall performance, and (ii) provide the closest approximation to actual searcher behaviour, although a fixed depth approach also performs remarkably well. Findings from this study have implications regarding how we build measures, and how we conduct simulations of search behaviours.

Never Gonna Give You Up Even When It Is Suboptimal

“…Overall, their resident time was close to the optimal resident time, although they tended to overstay, corresponding to the findings from previous studies (e.g., Hutchinson et al, 2008). In regard to the strategies used by the participants, most participants relied on the GUT strategy, while a minority was best described by the FT strategy, which, again, is in line with previous research (Hutchinson et al, 2008;Maxwell et al, 2015;Wilke et al, 2009). This suggests that separating the decision process from processing rewards did not change participants' strategies.…”

“…Thus, while it was not our intention, the design of Experiment 2 was also the best for pushing participants to use the FT over the GUT strategy. The preference for using the GUT strategy over the other strategies matches findings from previous studies (Hutchinson et al., 2008; Maxwell et al., 2015; Wilke et al., 2009), suggesting that although people are able to adjust their foraging strategy to the environment, that is, by staying longer when travel time between patches increases, they do not switch strategies. This deviates from research in other areas of decision‐making reporting that people adaptively switch between different strategies (Gigerenzer & Selten, 2002; Kolling & Akam, 2017; Payne et al., 1988).…”

“…In contrast, Zhang et al found that participants stayed longer in the current patch when the previous patch was rich compared to when the previous patch was poor. Indeed, on the strategy level, most studies find evidence that people rely on the GUT strategy (Hutchinson et al, 2008;Maxwell, Azzopardi, Järvelin, & Keskustalo, 2015;Wilke, Hutchinson, Todd, & Czienskowski, 2009). For example, Wilke et al (2009) explored cognitive foraging using a word scrabble task.…”

Report on the Information Retrieval Festival (IRFest2017)

“…how many items look relevant?). His poster compared different stopping strategies based on fixed and adaptive rules [17,18]. Experiments conducted on a large news collection for the task of ad-hoc topic search revealed that adaptive stopping strategies were more robust and lead to greater retrieval performance.…”

Commercialization of state-owned broadcast networks in China: a case study on the operation of Beijing Renmin Radio Station

“…Traditional simulation methods typically decompose a user's interactive search behavior into a series of independent steps, including submitting queries, browsing Search Engine Results Pages (SERPs), clicking results, reading and evaluating documents, and deciding when to stop [3]. Therefore, they require a dedicated simulation strategy to be designed for each step, such as generating search queries by extracting terms from language models associated with specific documents or topics [1,2,4,7,17], estimating the probability of a user clicking on search results based on historical data using click models [5,8,11,12,16], and deciding when a user stops searching based on a set of predefined simplistic assumptions through heuristic rules [10,19,[24][25][26]. However, these approaches fail to fully consider the dynamic and interdependent nature of user behavior, In particular, they often fail to model how some cognitive factors, such as the information needs and background knowledge, and cognitive processes, including the learning and reasoning, would affect and drive user's action.…”