Beyond topicality

Boyce, Bert R.

doi:10.1016/0306-4573(82)90033-4

Cited by 67 publications

(8 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The relevance of an item refers to its value with respect to a searcher's information need. We adopt the notion of topical relevance, the match between an item and the general topic of the request [4,13]. Specifically, we use binary topical relevance, where an item is considered relevant if any portion of the item-often a text document-would be valuable with respect to the information need.…”

Section: Relevancementioning

confidence: 99%

“…Such recall metrics for ranked retrieval systems come in two flavors. 4 The first flavor of recall metrics measures the fraction of relevant items found after a searcher terminates their scan of the ranked list. Metrics like R 1000 and RP use a model of search depth to simulate how deep a searcher will scan.…”

Section: Total Search Efficiencymentioning

confidence: 99%

Section: Possible Information Needsmentioning

confidence: 99%

“…Multiple authors describe topical relevance as necessary but not sufficient for psychological relevance [4,13,43]. Indeed, these papers suggest that, like multistage retrieval [32], topical relevance may be a good method to generate candidates that are then scanned for psychologically relevant items [4]. This separation of blatantly non-relevant items from possibly relevant items is similar to protocols used in inter-item preference evaluation which omit non-relevant items [7,10].…”

Section: Possible Information Needsmentioning

confidence: 99%

See 3 more Smart Citations

Recall as a Measure of Ranking Robustness

Díaz¹,

Mitra²

2023

Preprint

View full text Add to dashboard Cite

Researchers use recall to evaluate rankings across a variety of retrieval, recommendation, and machine learning tasks. While there is a colloquial interpretation of recall in set-based evaluation, the research community is far from a principled understanding of recall metrics for rankings. The lack of principled understanding of or motivation for recall has resulted in criticism amongst the retrieval community that recall is useful as a measure at all. In this light, we reflect on the measurement of recall in rankings from a formal perspective. Our analysis is composed of three tenets: recall, robustness, and lexicographic evaluation. First, we formally define 'recall-orientation' as sensitivity to movement of the bottom-ranked relevant item. Second, we analyze our concept of recall orientation from the perspective of robustness with respect to possible searchers and content providers. Finally, we extend this conceptual and theoretical treatment of recall by developing a practical preference-based evaluation method based on lexicographic comparison. Through extensive empirical analysis across 17 TREC tracks, we establish that our new evaluation method, lexirecall, is correlated with existing recall metrics and exhibits substantially higher discriminative power and stability in the presence of missing labels. Our conceptual, theoretical, and empirical analysis substantially deepens our understanding of recall and motivates its adoption through connections to robustness and fairness.

show abstract

Section: Relevancementioning

confidence: 99%

Section: Total Search Efficiencymentioning

confidence: 99%

Section: Possible Information Needsmentioning

confidence: 99%

Section: Possible Information Needsmentioning

confidence: 99%

See 2 more Smart Citations

Recall as a Measure of Ranking Robustness

Díaz¹,

Mitra²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Joudrey (2005) supports the distinction between extensional and intensional aboutness, and uses the terms aboutness and subject to distinguish the two (p. 59). Boyce (1982) uses the terms topicality and informativeness from a two-stage view of relevance and the retrieval process. While considered "operationally necessary" (p. 106), topicality is considered an insufficient condition for user based relevance.…”

Section: Aboutness As a Two-type Distinctionmentioning

confidence: 99%

The Life and Times of Aboutness: A Review of the Library and Information Science Literature

Rondeau¹

2014

EBLIP

View full text Add to dashboard Cite

the terms of the Creative Commons-AttributionNoncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. AbstractObjective -This paper explores the concept of "aboutness", its related terms, and the process of aboutness determination as found in the Library and Information Science Literature.Methods -A review of the literature pertaining to aboutness determination and related terms specific to Library and Information Science was undertaken, borrowing extensively from the literature review of Daniel Joudrey's (2005) dissertation, Building puzzles and growing pearls: A qualitative exploration of determining aboutness, as well as examining a small selection of research and articles not discussed by Joudrey. In addition, a concept map was developed to outline many of the concepts and theories found.Results -The LIS literature demonstrates conflicting positions surrounding the term aboutness and its correlates. Despite the lack of firm agreement on terminologies, the notion of subject is explored because it featured prominently in the literature. As well, intensional and extensional aboutness are explored in contrast to a more subjectivist perspective which asserts that a document's aboutness cannot be separated from its reader. Aboutness determination is also examined through the lens of theme and rheme, that is what is presupposed in a document versus what is new. Aboutness from the user perspective featured considerably in the literature, with many authors asserting both the Evidence Based Library and Information Practice 2014, 9.1 15 importance and the challenge of mediating knowledge on behalf of the user. The stage of aboutness determination in the complete process of subject indexing is also presented here, and the findings demonstrate that the stages are anything but linear and sequential.Conclusion -While the findings are more expositional than conclusive, they demonstrate the complexities and challenges surrounding the concept of "aboutness" and the process of its determination. The value of this review is in its ability to present the ways in which scholars and practitioners have attempted to grapple with this conundrum. Although indexers may find temporary solace in cataloguing manuals that outline aboutness determination procedures, underneath these "safe surfaces" is an ambiguous concept further complicated by obscure and incomplete processes. This review provides an opportunity to reflect on those challenges and to further the discussion.

show abstract

User-defined relevance criteria: An exploratory study

Barry

1994

J. Am. Soc. Inf. Sci.

354

300

View full text Add to dashboard Cite

The objective of this study was to describe the criteria mentioned by users evaluating the information within documents as it related to the users' information need situations. Data were collected by asking users in an academic environment to evaiu-ate representations and the full text of documents that had been retrieved specifically for each user's information need situation. Users were asked to mark the portions of the document representations or of the full text of documents that indicated to the users whether they would or would not pursue the information within documents. An open-ended interview technique was then employed to discuss each marked portion with users. The interviews were au-diotaped, the tapes transcribed, and the transcriptions were content analyzed in order to identify and describe evaluation criteria. The results indicate that the criteria employed by users included tangible characteristics of documents (e.g., the information content of the document, the provision of references to other sources of information), subjective qualities (e.g., agreement with the information provided by the document) and situational factors (e.g., the time constraints under which the user was working). The implications of this research for our understanding of the concept of relevance, and for the design and evaluation of information retrieval systems, are discussed. introduction The focus throughout the following discussion is on information retrieval systems that provide users with references to printed, textual materials (i.e., bibliographic retrieval systems). The discussion does not encompass situations in which users are searching for known items, or situations in which users are routinely provided with references as part of a selective dissemination of information system. Rather, the focus is on the user who can present a request for unknown or unfamiliar information, and purposively approaches the system (or the system intermediary) in an attempt to discover such information. This exploratory study was based on the assumption that, although the retrieval mechanism employed by information systems is based primarily on subject matching to retrieve topically appropriate documents, motivated users examining information as it relates to current information need 0 1994 John Wiley & Sons, Inc. situations evaluate that information based on criteria beyond the topical appropriateness of documents. This research was an attempt to identify and describe those criteria. Throughout the following discussion, there are many references to users' "information need situations." In the context of this research, an information need situation is very simply defined as some situation in which a user finds himself or herself, in which there is a need for information. This is not meant to imply that there is such a phenomenon as an "information need" which is static and unchanging. Rather, this research is based on the assumption that such an information need situation encompasses all factors that the user brings to the situatio...

show abstract

Beyond topicality

Cited by 67 publications

References 7 publications

Recall as a Measure of Ranking Robustness

Recall as a Measure of Ranking Robustness

The Life and Times of Aboutness: A Review of the Library and Information Science Literature

User-defined relevance criteria: An exploratory study

Contact Info

Product

Resources

About