Toward an understanding of the dynamics of relevance judgment: An analysis of one person's search behavior

Tang, Rong; Solomon, Paul

doi:10.1016/s0306-4573(97)00081-2

Cited by 105 publications

(90 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…When revising knowledge structure, a person's conceptual construct about the topic is vague. Therefore, she has difficulty in recognizing useful sources, leading to loose relevance criteria [1,[25][26][27]. Many of the sources may be useful, but she is not able to tell which ones, because she has as yet no firm ideas on how to shape her topic.…”

Section: Learning In the Search Processmentioning

confidence: 99%

Searching as learning: A systematization based on literature

Vakkari

2016

Journal of Information Science

130

101

View full text Add to dashboard Cite

The paper surveys empirical studies on the relations between information searching and learning, and presents some reflections about learning in a search process based on the findings. First, the meaning of the concepts 'learning' and 'searching' is briefly defined. Learning is conceptualized as changes in one's knowledge structures. Then it is described more in detail how learning occurs in the search process. The point of departure is to focus on tasks that require the restructuring of knowledge structures and to analyse how gradual stabilization of those structures is related to accessing and interacting with information sources. After that, empirical studies on searching and learning are categorized by identifying independent and dependent variables in those studies. In conclusion, some general remarks on the topic are presented.

show abstract

Section: Learning In the Search Processmentioning

confidence: 99%

Searching as learning: A systematization based on literature

Vakkari

2016

Journal of Information Science

130

101

View full text Add to dashboard Cite

show abstract

“…Relevance judgments have also been studied as part of the overall information seeking process. Again, the focus has often been on the criteria by which users judge the relevance of a document to a task [4,9] but much of this work has also investigated how a user's conception of relevance changes over time, as the search process develops [4,6,9]. The effect of relevance level (i.e.…”

Section: Previous Workmentioning

confidence: 99%

“…Relevance judgment is important to both the search process itself [4], and in the creation of test collections [3,5]. With regard to the latter, there is a considerable body of work which has investigated the criteria assessors use to judge relevance for the creation of text collections [3,5].…”

Section: Introductionmentioning

confidence: 99%

Is relevance hard work?

Villa

Halvey

2013

Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

The judging of relevance has been a subject of study in information retrieval for a long time, especially in the creation of relevance judgments for test collections. While the criteria by which assessors' judge relevance has been intensively studied, little work has investigated the process individual assessors go through to judge the relevance of a document. In this paper, we focus on the process by which relevance is judged, and in particular, the degree of effort a user must expend to judge relevance. By better understanding this effort in isolation, we may provide data which can be used to create better models of search. We present the results of an empirical evaluation of the effort users must exert to judge the relevance of document, investigating the effect of relevance level and document size. Results suggest that "relevant" documents require more effort to judge when compared to highly relevant and not relevant documents, and that effort increases as document size increases.

show abstract

“…They found that typicality, emotion and aesthetic appearance were the three most important criteria, applied across all three tasks, where typicality was deemed the most important criterion for all three tasks (according to the authors, typicality is a criterion that can exhibit universal representation of an object in a photo). Lancaster (1979); Barry (1994); Barry and Schamber (1998);Mizzaro (1997); Choi and Rasmussen (2002) and Tang and Solomon (1998) reported that the end-users of IR systems decided the relevance of retrieved documents based upon their particular information needs.…”

Section: Image Relevance Studiesmentioning

confidence: 99%

A study on the relevance criteria for medical images

Sedghi¹,

Sanderson²,

Clough³

2008

Pattern Recognition Letters

View full text Add to dashboard Cite

This paper reports the results of a study investigating the relevance criteria used by health care professionals when seeking medical images. Data was collected from 26 participants using a think-a-loud protocol and face-to-face interviews and analysed using the Straussian version of Grounded Theory (GT). Findings show that participants made use of 27 relevance criteria, although did not agree on the most important. Our findings suggest that users apply different criteria in different situations when evaluating the relevancy of medical images. In addition, we have investigated the coverage of relevance criteria to search statements from the medical track of ImageCLEF (ImageCLEFMed). Analysis indicates that some of the criteria identified by our participants could be included in future runs of the international evaluation campaign.

show abstract

Toward an understanding of the dynamics of relevance judgment: An analysis of one person's search behavior

Cited by 105 publications

References 16 publications

Searching as learning: A systematization based on literature

Searching as learning: A systematization based on literature

Is relevance hard work?

A study on the relevance criteria for medical images

Contact Info

Product

Resources

About