Relevance as an aid to evaluation in OPACs

Abstract

The relevance of retrieved documents or document descriptions has been a central measurement in the evaluation of information retrieval (IR) systems. Online public access catalogues (OPACs) are sirrular in many ways and so relevance should be an appropnate evaluation tool in measuring their performance. The development of relevance in evaluating IR systems is described and also the important differences between such systems and OPACs. Characteristics of the endusers of both systems are discussed with the conclusion that end-users' motivation and behaviour are as yet not fully understood. The implication of this is that relevance is a central feature, but only one, of an OPAC user's overall information need.

A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments

“…“The most common forms of evaluation of OPACs have been (1) surveys (2) comparative studies, and (3) transaction log analysis” (O’Brien, 1990, p. 269). One might assume this quotation would now be outdated, but it actually does not appear to be.…”

“…Studies on the retrieval effectiveness of a particular information system typically involve assessing the relevance of documents for a search query or information need. Relevance assessments are commonly used for information retrieval (IR) evaluation, particularly within TREC (Text REtrieval Conference), whereas relevance as a "fundamental concern" for OPAC evaluation had been ascertained decades ago (O'Brien, 1990). Although there are some studies on testing the retrieval effectiveness of OPACs or discovery systems, there are no consistent criteria for evaluation, as they are met within TREC.…”