This article summarizes the theory of psychological relevance proposed by Dan Sperber and Deirdre Wilson (1988), to explicate the relevance of speech utterances to hearers in everyday conversation.The theory is then interpreted as the concept of relevance in information retrieval, and an extended example is presented. Implications of psychological relevance for research in information retrieval; evaluation of information retrieval systems; and the concepts of information, information need, and the information-seeking process are explored. Connections of the theory to ideas in bibliometrics are also suggested.

No abstract

The purpose of this article is to bring attention to the problem of variations in relevance assessments and the effects that these may have on measures of retrieval effectiveness. Through an analytical review of the literature, I show that despite known wide variations in relevance assessments in experimental test collections, their effects on the measurement of retrieval performance are almost completely unstudied. I will further argue that what we know about the many variables that have been found to affect relevance assessments under experimental conditions, as well as our new understanding of psychological, situational, user-based relevance, point to a single conclusion. We can no longer rest the evaluation of information retrieval systems on the assumption that such variations do not significantly affect the measurement of information retrieval performance.A series of thorough, rigorous, and extensive tests is needed, of precisely how, and under what conditions, variations in relevance assessments do, and do not, affect measures of retrieval performance. We need to develop approaches to evaluation that are sensitive to these variations and to human factors and individual differences more generally. Our approaches to evaluation must reflect the real world of real users.

The problem studied in this research is that of developing a set of formal statistical rules for the purpose of identifying the keywords of a document‐words likely to be useful as index terms for that document. The research was prompted by the observation, made by a number of writers, that non‐specialty words, words which possess little value for indexing purposes, tend to be distributed at random in a collection of documents. In contrast, specialty words are not so distributed. In Part I of the study, a mixture of two Poisson distributions is examined in detail as a model of specialty word distribution, and formulas expressing the three parameters of the model in terms of empirical frequency statistics are derived. The fit of the model is tested on an experimental document collection and found to be acceptable for the purposes of the study. A measure intended to identify specialty words, consistent with the 2‐Poisson model, is proposed and evaluated.

No abstract

In Part I of this study,* a mixture of two Poisson distributions was examined as a model of specialty word distribution. Formulas expressing the three parameters of the model in terms of empirical frequency statistics were derived, and a statistical measure intended to identify specialty words, consistent with the model, was proposed. In the present paper, Part II of the study, a probabilistic model of keyword indexing is outlined, and some of the consequences of the model are examined. An algorithm defining a measure of indexability is developed‐a measure intended to reflect the relative significance of words in documents. The measure is evaluated and is found to consistently produce indexes superior to those produced by another measure which had previously been identified in the literature as producing the best results.

This study1 assesses the ways in which citation searching of scholarly print journals is and is not analogous to backlink searching of scholarly e‐journal articles on the WWW, and identifies problems and issues related to conducting and interpreting such searches. Backlink searches are defined here as searches for Web pages that link to a given URL. Backlink searches were conducted on a sample of 39 scholarly electronic journals. Search results were processed to determine the number of backlinking pages, total backlinks, and external backlinks made to the e‐journals and to their articles. The results were compared to findings from a citation study performed on the same e‐journals in 1996. A content analysis of a sample of the files backlinked to e‐journal articles was also undertaken. The authors identify a number of reliability issues associated with the use of “raw” search engine data to evaluate the impact of electronic journals and articles. No correlation was found between backlink measures and ISI citation measures of e‐journal impact, suggesting that the two measures may be assessing something quite different. Major differences were found between the types of entities that cite, and those that backlink, e‐journal articles, with scholarly works comprising a very small percentage of backlinking files. These findings call into question the legitimacy of using backlink searches to evaluate the scholarly impact of e‐journals and e‐journal articles (and by extension, e‐journal authors).

No abstract