Regions and levels: Measuring and mapping users' relevance judgments

Spink, Amanda; Greisdorf, Howard

doi:10.1002/1097-4571(2000)9999:9999<::aid-asi1564>3.0.co;2-l

Cited by 47 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, a substantial number of empirical studies (e.g. Barry, 1994;Cool, Belkin, Frieder, & Kantor, 1993;Park, 1993;Schamber, 1991;Spink & Greisdorf, 2001;Wang & Soergel, 1999) have revealed that people use much more diverse criteria than mere topicality to make relevance judgments in the traditional information retrieval environment. This study will Rieh 2 take these findings a step further by focusing on two factors which appear consistently across the previous studies: quality and authority.…”

Section: Introductionsupporting

confidence: 86%

Judgment of information quality and cognitive authority in the Web

Rieh¹

2002

J. Am. Soc. Inf. Sci.

268

292

View full text Add to dashboard Cite

In the Web, making judgments of information quality and authority is a difficult task for most users because overall, there is no quality control mechanism. This study examines the problem of the judgment of information quality and cognitive authority by observing people's searching behavior in the Web. Its purpose is to understand the various factors that influence people's judgment of quality and authority in the Web, and the effects of those judgments on selection behaviors. Fifteen scholars from diverse disciplines participated, and data were collected combining verbal protocols during the searches, search logs, and post-search interviews. It was found that the subjects made two distinct kinds of judgment: predictive judgment and evaluative judgment. The factors influencing each judgment of quality and authority were identified in terms of characteristics of information objects, characteristics of sources, knowledge, situation, ranking in search output, and general assumption. Implications for Web design which will effectively support people's judgments of quality and authority are also discussed. IntroductionOne of the advantages of searching in the Web is its grant of access to a great amount and a wide variety of information. As a result, however, people need some ways to reduce the large amount of information in order to select the information that they want. In traditional information retrieval, this problem has long been discussed within the context of "topical relevance"; that is, in terms of whether the topic of the query matches the topic of a document. However, a substantial number of empirical studies (e.g. Barry, 1994;Cool, Belkin, Frieder, & Kantor, 1993;Park, 1993;Schamber, 1991;Spink & Greisdorf, 2001;Wang & Soergel, 1999) have revealed that people use much more diverse criteria than mere topicality to make relevance judgments in the traditional information retrieval environment. This study will Rieh 2 take these findings a step further by focusing on two factors which appear consistently across the previous studies: quality and authority. These two factors were chosen because it is believed that they may be more important relevance criteria than any other criteria identified in the previous studies, especially in a large uncontrolled environment, such as the Web.The concepts of quality and authority are not new. On the one hand, a number of studies of relevance criteria, particularly in the 1990s, identified various aspects of both concepts including "goodness" (Cool et al.), "usefulness" (Cool et al.), "accuracy/validity" (Barry), "recency" (Barry; Wang & Soergel), "perceived quality (Park), "actual quality" (Wang & Soergel), "expected quality" (Wang & Soergel), "authority" (Cool et al.; Wang & Soergel), and "reliability" (Schamber). On the other hand, in recent years, the notions of quality and authority have been discussed with respect to evaluation criteria of Web pages by examining different approaches and implementations. Librarians and researchers in library and information science (e....

show abstract

Section: Introductionsupporting

confidence: 86%

Judgment of information quality and cognitive authority in the Web

Rieh¹

2002

J. Am. Soc. Inf. Sci.

268

292

View full text Add to dashboard Cite

show abstract

“…Task has been shown to affect users' information seeking behaviors and relevance judgments in a variety of ways and is a good candidate variable for understanding more about search systems and user behavior [163,170,171,249,266,281]. Vakkari [279] provides an overview of task-based information searching.…”

Section: Information Needs: Tasks and Topicsmentioning

confidence: 99%

Methods for Evaluating Interactive Information Retrieval Systems with Users

Kelly

2007

FNT in Information Retrieval

394

308

View full text Add to dashboard Cite

“…The Search engine system views relevance of search engine results as "a result of the thinking that relevance is mostly affected by the internal aspects and manipulations of the system" [18]. Spink and Greisdorf [20] concluded that search results are improved when several levels Downloaded by [New York University] at 21:04 15 May 2015 of relevance are defined, rather than a binary measure (relevant vs. nonrelevant). Levels of relevance have already been adopted to measure the precision of Web search engines [6,8,14].…”

Section: Literature Reviewmentioning

confidence: 99%

Quality of search engines: how good is Google pagerank?

2010

Journal of Information and Optimization Sciences

View full text Add to dashboard Cite

Owing to the increasing use of the Internet for diverse applications, information retrieval has become a major research issue. Google, the largest search engine worldwide, utilizes its own PageRank algorithm to calculate the relevance of websites. This study proposes a practical method based on Information Retrieval Ratios (the Recall, Precision and Available ratios) to evaluate the validity of PageRank for measuring search engine quality. Analytical results reveal that correlations between PageRanks and information retrieval ratios in the search results of web pages in Traditional Chinese are not significant ( p > 0.05 in each case). This study concludes that PageRank is not a useful indicator of the relevance of results in Traditional Chinese search engines. Therefore, Google needs to enhance its PageRank algorithm.

show abstract

Regions and levels: Measuring and mapping users' relevance judgments

Cited by 47 publications

References 27 publications

Judgment of information quality and cognitive authority in the Web

Judgment of information quality and cognitive authority in the Web

Methods for Evaluating Interactive Information Retrieval Systems with Users

Quality of search engines: how good is Google pagerank?

Contact Info

Product

Resources

About