Studying human judgments of relevance

Abstract

This paper discusses the ways participants in a two-year ethnographic study judged relevance when engaged in searching and research tasks. Two experienced academics have been observed evaluating informative artefacts (documents, citations or other representations) encountered in the course of their own research projects. This study sought to explore the criteria and clues used to make decisions about the relevance of retrievable items. In presenting some of the findings from this longitudinal study, the paper demonstrates the value of this approach for enhancing our understanding of the evolving nature of human relevance judgments. The paper will describe how this interaction involves not only the notion of searcher-system communication, but a range of encounters that inform and influence that particular communication at the search interface. The paper suggests future collaboration between system specialists and human behaviour specialists to further our understanding of the socio-material systems in which people make judgments of relevance. Keywordshuman relevance judgments, nature of relevance in contexts, taskbased information behaviour CONCEPTUAL FRAMEWORKRelevance is a central but much debated concept in Information Science. After more than fifty years of discussion, there is still a lack of consensus about the meaning of the concept. As a result, a number of researchers over the years have observed that, despite its significance, it remains one of the least understood concepts of information retrieval [e.g. 3, 4, 5, 6].Considerable effort has gone into finding ways to relate searcher perspectives of relevance to those embodied in information systems. From a searcher's perspective, relevance assessment is a process by which she constantly shapes, defines and refines searching. This view of relevance is not contained in a system-

High-Dimensional Image Data Sets Retrieval: Improving Accuracy Using a Weighted Relevance Feedback

“…Indeed, there are a number of perceptual stimuli that are more likely than others to get our attention by distracting the rest. In addition, the brain is not able to e®ectively track multiple tasks at once, but it knows to move very quickly from one task to another or remember a primary task while others are conducted [24]. More speci¯cally, in the context of interaction between the brain and images, the intention of¯nding a certain class of images is strongly related to consciousness.…”

The impact of document structure on keyphrase extraction

“…The XML documents were parsed and the textual content of each XML element was indexed using Lucene. 1 (as separate fields, with stemming and without stopword removal). In this way elements can be searched, and we can also retrieve the full text content of a specified element.…”

“…the clues that researchers use to make relevance decisions, and have been found beneficial for exploring digital libraries [1,5]. As manual assignment of keyphrases is a tedious process, various methods to automatically suggest keyphrases have been proposed [13], that select phrases based on features capturing usage of the phrase.…”

DILIA - A DIGITAL LIBRARY ASSISTANT - A New Approach to Information Discovery through Information Extraction and Visualization

“…), publishing date (e.g., year), and its text. These attributes can contain specific words, i.e., terms that can be recognized by the information seeker as relevant and trigger the formulation of refined search queries (Barry, 1994;Anderson, 2006).…”

“…Studies conducted by (Anderson, 2006) reported that it was difficult to find and specify appropriate terms to define more precise search queries, especially, if an information seeker was unfamiliar with the terminology of the problem domain, or if this terminology changed over time.…”