User-defined relevance criteria: An exploratory study

Abstract

The objective of this study was to describe the criteria mentioned by users evaluating the information within documents as it related to the users' information need situations. Data were collected by asking users in an academic environment to evaiu-ate representations and the full text of documents that had been retrieved specifically for each user's information need situation. Users were asked to mark the portions of the document representations or of the full text of documents that indicated to the users whether they would or would not pursue the information within documents. An open-ended interview technique was then employed to discuss each marked portion with users. The interviews were au-diotaped, the tapes transcribed, and the transcriptions were content analyzed in order to identify and describe evaluation criteria. The results indicate that the criteria employed by users included tangible characteristics of documents (e.g., the information content of the document, the provision of references to other sources of information), subjective qualities (e.g., agreement with the information provided by the document) and situational factors (e.g., the time constraints under which the user was working). The implications of this research for our understanding of the concept of relevance, and for the design and evaluation of information retrieval systems, are discussed. introduction The focus throughout the following discussion is on information retrieval systems that provide users with references to printed, textual materials (i.e., bibliographic retrieval systems). The discussion does not encompass situations in which users are searching for known items, or situations in which users are routinely provided with references as part of a selective dissemination of information system. Rather, the focus is on the user who can present a request for unknown or unfamiliar information, and purposively approaches the system (or the system intermediary) in an attempt to discover such information. This exploratory study was based on the assumption that, although the retrieval mechanism employed by information systems is based primarily on subject matching to retrieve topically appropriate documents, motivated users examining information as it relates to current information need 0 1994 John Wiley & Sons, Inc. situations evaluate that information based on criteria beyond the topical appropriateness of documents. This research was an attempt to identify and describe those criteria. Throughout the following discussion, there are many references to users' "information need situations." In the context of this research, an information need situation is very simply defined as some situation in which a user finds himself or herself, in which there is a need for information. This is not meant to imply that there is such a phenomenon as an "information need" which is static and unchanging. Rather, this research is based on the assumption that such an information need situation encompasses all factors that the user brings to the situatio...

Query-Sensitive Similarity Measures for Information Retrieval

“…Research into the concept of relevance has indicated that topicality plays a significant role in the determination of relevance, although it does not automatically result in relevance for users (Barry, 1994).…”

DILIA - A DIGITAL LIBRARY ASSISTANT - A New Approach to Information Discovery through Information Extraction and Visualization

“…), publishing date (e.g., year), and its text. These attributes can contain specific words, i.e., terms that can be recognized by the information seeker as relevant and trigger the formulation of refined search queries (Barry, 1994;Anderson, 2006).…”

A study on the relevance criteria for medical images

“…They found that typicality, emotion and aesthetic appearance were the three most important criteria, applied across all three tasks, where typicality was deemed the most important criterion for all three tasks (according to the authors, typicality is a criterion that can exhibit universal representation of an object in a photo). Lancaster (1979); Barry (1994); Barry and Schamber (1998);Mizzaro (1997); Choi and Rasmussen (2002) and Tang and Solomon (1998) reported that the end-users of IR systems decided the relevance of retrieved documents based upon their particular information needs.…”