Regions and levels: Measuring and mapping users' relevance judgments

Spink, Amanda; Greisdorf, Howard

doi:10.1002/1097-4571(2000)9999:9999<::aid-asi1564>3.3.co;2-c

Cited by 23 publications

(18 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also examined whether three coarse categories for relevance may be better than four, but found little evidence for this. What the "optimal" number of relevance grades should be, is still an open problem for further research, although we have evidence that three or four is a sensible choice, which is also inline with the findings of previous work (Gwizdka, 2014;Spink & Greisdorf, 2001;Sormunen, 2002;Vakkari, Luoma, & P€ ontinen, 2014).…”

Section: Discussionsupporting

confidence: 85%

“…The students were guided that the order of the items in the list presented to them has no relation to their relevance to the query. We were motivated by the study of Spink and Greisdorf (2001), who assert that there is no consensus regarding how relevance judgments should be measured and thus no specific instructions on this matter can be provided to the users. We also adopted Spink and Greisdorf's (2001) four-point scale for relevance judgments.…”

Section: Study Setupmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of change in users' assessment of search results over time

Zhitomirsky‐Geffet

Bar‐Ilan

Levene

2016

Asso for Info Science & Tech

View full text Add to dashboard Cite

We present the first systematic study of the influence of time on user judgements for rankings and relevance grades of web search engine results. The goal of this study is to evaluate the change in user assessment of search results and explore how users' judgements change. To this end, we conducted a large-scale user study with 86 participants who evaluated two different queries and four diverse result sets twice with an interval of two months. To analyse the results we investigate whether two types of patterns of user behaviour from the theory of categorical thinking hold for the case of evaluation of search results:(1) coarseness and (2) locality. To quantify these patterns we devised two new measures of change in user judgements and distinguish between local (when users swap between close ranks and relevance values) and non-local changes. Two types of judgements were considered in this study: 1) relevance on a 4-point scale, and 2) ranking on a 10-point scale without ties. We found that users tend to change their judgements of the results over time in about 50% of cases for relevance and in 85% of cases for ranking.However, the majority of these changes were local.2

show abstract

Section: Discussionsupporting

confidence: 85%

Section: Study Setupmentioning

confidence: 99%

Analysis of change in users' assessment of search results over time

Zhitomirsky‐Geffet

Bar‐Ilan

Levene

2016

Asso for Info Science & Tech

View full text Add to dashboard Cite

show abstract

“…Users may find that some results precisely match their needs, while some results match their needs only to a certain extent, and others fail to match completely. However, systems typically collapse results into two sets, in which one set combines partially relevant items with highly relevant items, and the other set consists of non-relevant items (Spink & Greisdorf, 2001). Thus, matching measurements such as precision and recall are based on these dual sets rather than sets based on the regions of relevance.…”

Section: Related Studiesmentioning

confidence: 99%

“…Wallis and Thom (1996) stress that it is not simple to convey relevance in terms of degrees but emphasize the need to retrieve material that is both relevant and partially relevant. Binary assessments hide the variability, complexity and continuity of relevance Maglaughlin & Sonnenwald, 2002;Spink & Greisdorf, 2001). Indeed, conflating relevance for the purposes of IR evaluation can lead to skewed results since relevance inherently consists of a range of degrees.…”

Section: Related Studiesmentioning

confidence: 99%

An approach to document clustering based on system relevance

Desai

Spink

2004

Proc of Assoc for Info

Self Cite

View full text Add to dashboard Cite

Search engines fail to make a clear distinction between items of varying relevance when presenting search results to users. rely on the user of the system to estimate which items are relevant, partially relevant, or not relevant. The user of the system is given the tedious task of distinguishing between documents that are relevant to different degrees. This often hinders the accessibility of relevant or partially rerevant documents, particularly when the results set is large and many non-relevant documents are scattered throughout the set. In this paper, we present the results of a clustering scheme that groups documents within relevant, partially relevant, and not relevant clusters for a given search. A ranking algorithm accomplishes the task of clustering the documents based on system relevance. Data was collected from endusers issuing categorical, interval, and descriptive relevance judgments. The degree of overlap between users and the system for each of the clustered regions was measured. This research showed that clustering documents on the Web by regions of relevance is quite feasible.

show abstract

“…Schamber et al 7 reviewed the definition of relevance and concluded that relevance is a multi-dimensional cognitive concept, dependent on users' perceptions of information and their own information needs, dynamic (as quality judgements may change), and complex, but measurable if the user's perspective is the focus of investigation. Other researchers have considered multiple levels of relevance, 8 partial relevance, [9][10][11] and changing frameworks for relevance depending on the stage of problem solving. 12 Research has also focused more on the usefulness to the user; the value of the information retrieved and exam-ined quality attributes such as goodness, usefulness, currency, accuracy, and trustworthiness.…”

Section: Introductionmentioning

confidence: 99%

Beyond relevance and recall: testing new user‐centred measures of database performance

Stokes

Foster

Urquhart

2009

Health Info Libraries J

View full text Add to dashboard Cite

Background: Measures of the effectiveness of databases have traditionally focused on recall, precision, with some debate on how relevance can be assessed, and by whom. New measures of database performance are required when users are familiar with search engines, and expect full text availability. Objectives: This research ascertained which of four bibliographic databases (BNI, CINAHL, MEDLINE and EMBASE) could be considered most useful to nursing and midwifery students searching for information for an undergraduate dissertation. Methods: Searches on title were performed for dissertation topics supplied by nursing students (n ¼ 9), who made the relevance judgements. Measures of recall and precision were combined with additional factors to provide measures of effectiveness, while efficiency combined measures of novelty and originality and accessibility combined measures for availability and retrievability, based on obtainability.Results: There were significant differences among the databases in precision, originality and availability, but other differences were not significant (Friedman test). Odds ratio tests indicated that BNI, followed by CINAHL were the most effective, CINAHL the most efficient, and BNI the most accessible. Conclusions:The methodology could help library services in purchase decisions as the measure for accessibility, and odds ratio testing helped to differentiate database performance.

show abstract

Regions and levels: Measuring and mapping users' relevance judgments

Cited by 23 publications

References 26 publications

Analysis of change in users' assessment of search results over time

Analysis of change in users' assessment of search results over time

An approach to document clustering based on system relevance

Beyond relevance and recall: testing new user‐centred measures of database performance

Contact Info

Product

Resources

About