Relevance judging, evaluation, and decision making in virtual libraries: A descriptive study

Human information-seeking behavior is complicated. Activity theory is a powerful theoretical instrument to untangle the "complications." Based on activity theory, a comprehensive framework is proposed in Part I (Y. Xu, 2007) of this report to describe interactive information retrieval (IIR) behavior. A set of propositions is also proposed to describe the mechanisms governing users' cognitive activity and the interaction between users' cognitive states and manifested retrieval behavior. An empirical study is carried out to verify the propositions. The authors' experimental simulation of 81 participants in one search session indicates the propositions are largely supported. Their findings indicate IIR behavior is planned. Users adopt a divide-and-conquer strategy in information retrieval. The planning of information retrieval activity is also partially manifested in query revision tactics. Users learn from previously read documents. A user's interaction with a system ultimately changes the user's information need and the resulting relevance judgment, but the dynamics of topicality perception and novelty perception occur at different paces.

“…Usefulness is not a relevance criterion, but an overall measure of situational relevance (Fitzgerald & Galloway, 2001). Topicality and novelty are relevance criteria (Xu & Chen, 2006).…”

Section: The Corpus Search Engine and Activitymentioning

confidence: 99%

The dynamics of interactive information retrieval, Part II: An empirical study from the activity theory perspective

Liu

2007

“…Next, our research assistant explained the definition of topicality, novelty, and relevance in layman's terms. Relevance was explained as usefulness as suggested by past studies (e.g., Fitzgerald & Galloway, 2001). Specifically, we stated in the experiment instruction:…”

Section: Experiments Proceduresmentioning

confidence: 99%

“…Researchers have repeatedly found that users' relevance judgment encompasses not only a topical match between an information need and a document but also the novelty, understandability, reliability, and scope of the document (Bateman, 1998;Fitzgerald & Galloway, 2001;Wang & Soergel, 1998;Xu & Chen, 2006). Among all these criteria, novelty judgments have received particular interest by system-centered researchers (Allan, Wade, & Bolivar, 2003;Brants, Chen, & Farahat, 2003;Kumaran & Allan, 2004;Yang, Zhang, Carbonell, & Jin, 2002;Zhang, Callan, & Minka, 2002).…”

Section: Introductionmentioning

confidence: 95%

Novelty and topicality in interactive information retrieval

Yin²

2007

The information science research community is characterized by a paradigm split, with a system-centered cluster working on information retrieval (IR) algorithms and a user-centered cluster working on user behavior. The two clusters rarely leverage each other's insight and strength. One major suggestion from user-centered studies is to treat the relevance judgment of documents as a subjective, multidimensional, and dynamic concept rather than treating it as objective and based on topicality only. This study explores the possibility to enhance users' topicality-based relevance judgment with subjective novelty judgment in interactive IR. A set of systems is developed which differs in the way the novelty judgment is incorporated. In particular, this study compares systems which assume that users' novelty judgment is directed to a certain subtopic area and those which assume that users' novelty judgment is undirected. This study also compares systems which assume that users judge a document based on topicality first and then novelty in a stepwise, noncompensatory fashion and those which assume that users consider topicality and novelty simultaneously and as compensatory to each other. The user study shows that systems assuming directed novelty in general have higher relevance precision, but systems assuming a stepwise judgment process and systems assuming a compensatory judgment process are not significantly different.

“…• Fitzgerald and Galloway (2001) observed 10 undergraduate students using a digital library for their projects in assessing 138 retrieved documents to derive relevance-and evaluation-related reasoning. They identified 11 relevance and 11 evaluation categories of reasoning, both entering in relevance decisions.…”

Section: Relevance Clues What Makes Information or Information Objectmentioning

confidence: 99%

Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance

Saračević

2007

209

214

Relevance is a, if not even the, key notion in information science in general and information retrieval in particular. This two-part critical review traces and synthesizes the scholarship on relevance over the past 30 years or so and provides an updated framework within which the still widely dissonant ideas and works about relevance might be interpreted and related. It is a continuation and update of a similar review that appeared in 1975 under the same title, considered here as being Part I. The present review is organized in two parts: Part II addresses the questions related to nature and manifestations of relevance, and Part III addresses questions related to relevance behavior and effects. In Part II, the nature of relevance is discussed in terms of meaning ascribed to relevance, theories used or proposed, and models that have been developed. The manifestations of relevance are classified as to several kinds of relevance that form an interdependent system of relevancies. In Part III, relevance behavior and effects are synthesized using experimental and observational works that incorporated data. In both parts, each section concludes with a summary that in effect provides an interpretation and synthesis of contemporary thinking on the topic treated or suggests hypotheses for future research. Analyses of some of the major trends that shape relevance work are offered in conclusions. Prologue to Part III: How It Is Connected and What This Work Is All AboutTo provide a continuation from the preceding Part II, a few basic descriptions about this work are repeated.As stated in the Preface to Part II, in 1975 I published a review about relevance under the same title, without, of course, "Part I" in the title (Saracevic, 1975). There was no plan then to have another related review 30 years later-but things happen. The intent of the 1975 work was "to explore the meaning of relevance as it has evolved in information science and to provide a framework within which various interpretations of relevance can be related" (Saracevic, 1975, p. 321).Building on the examination of relevance in the preceding (1975) review, this work (2007) follows the travails of relevance in information science for the past 30 years. It is an update. The aim of this work is still substantially the same: It is an attempt to trace the evolution of thinking on relevance in information science for the past three decades and to provide an updated, contemporary framework within which the still widely dissonant ideas on relevance might be interpreted and related to one another.The organization of the present review, offered in two parts, addresses the questions related to relevance nature, manifestations, behavior, and effects. Following the Introduction and a Historical Footnote, the preceding Part II (this issue, pp. 1915-1933) started with a general section on the nature of relevance by synthesizing its meanings, following with sections on theories and models of relevance that are, in effect, further elaborations on the nature of relevance. Part ...