The size of retrieval sets

Abstract

This study was concerned with the question of how much information users of online search services want when they have a search conducted. This question was investigated within the context of multiattribute decision making. Thirty-two individuals rated their likely satisfaction with the results of hypothetical online searches on a Pl-point scale. The hypothetical searches varied in terms of the number of documents retrieved, the percentage of retrieved documents that were relevant to the searcher's topic (the precision of the search), and the cost of the search. These three attributes were combined factorially to construct the hypothetical searches. Each subject in the study rated all of the searches. The results showed that the precision of the search had the largest effect on the satisfaction ratings. The size of the retrieval set did not affect the ratings directly, but did combine with precision in a multiplicative fashion, in that high precision was not as highly valued when there were larger numbers of documents in the retrieval set. The results were interpreted in terms of a psychological theory called "information integration theory." How Much InformationIs Enough?Several years ago, Marcia Bates (1984) introduced the notion of the "fallacy of the perfect 30-item search" to the information retrieval (IR) literature. She claimed that clients and searchers can become too focused on the number of items retrieved in an online search, rather than on whether the items in the final retrieval set adequately address the user's query. While it is likely that search intermediaries do sometimes fall victim to this fallacy, it is less clear that it is a fallacy for the clients to want to put a limit on the number of documents they are willing to deal with, even if all the documents are relevant to their topic (Wiberley & Daugherty, 1988). After all, the end user presumably Received August 20, 1990; revised February 6, 1991; accepted March 12, 1991. 0 1992 has to read and understand the material retrieved by a search, whereas a search intermediary does not.Very often, designers and evaluators of information retrieval systems have ignored end users' possible aversions to information overload. For example, Wallace, Boyce, and Kraft (1988) modeled the cost of displaying additional records during a search strictly in terms of the monetary costs, ignoring such psychological costs as information overload. Furthermore, the two most common measures of IR system performance, recall and precision, are both percentage measures, which makes them insensitive to the actual number of documents in question. For instance, it's not at all clear that clients would be as satisfied with 80% precision in a loo-item search as they would with 80% precision in a 30-item search. Yet, identifying both simply as searches with 80% precision treats them as equally valuable. In fact, since it retrieves a greater number of relevant documents, the loo-item search would probably be considered the better of the two, because of its higher recall. Howev...

Name collection by ph.d. history students: inducing expertise

“…For example, Harter (1992, p. 608) indicates that if the user is using a “psychological” rather than purely “topical” criterion of relevance, a citation that “topically” matches the user's topic, but contains information the user has previously processed, may be passed over by the user because it is not “new” (cf. also, Kinnucan, 1992; Korfhage, 1997; Park, 1994; cf. Tiamiyu & Ajiferuke, 1988, for the related issue of interaction effects in judging citation lists).…”

Visualization in information retrieval: a three-level analysis

“…Concerns with the size of retrieval sets led to the investigation of whether the size of retrieval sets affects relevance judgement [57]. The findings from Kinnucan's study indicated that the size of retrieval sets affects relevance judgement in a complex way.…”

The Online Catalog: From Technical Services to Access Service

“…As for how many records are wanted, a 30-record response is widely considered a typical optimal result, and professional searchers view 50 to 70 references as optimal (Bates, 1986;Wiberly & Dougherty, 1988). Kinnucan (1992) found satisfaction relatively una ected by retrieval set size but sensitive to precision, though less so as retrieved set size increased.…”