An analysis of crowd workers mistakes for specific and complex relevance assessment task

Anderton, Jesse; Bashir, Maryam; Pavlu, Virgil; Aslam, Javed A.

doi:10.1145/2505515.2507884

Cited by 4 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We found no correlation between judging time and topic or document length (measured as number of chars or words, or with the ari readability index (Senter and Smith 1967)). This is consistent with the findings of (Anderton et al 2013) who looked at these dimensions within the TREC 2012 Crowdsourcing track collection. We also did not observe correlation of time spent with relevance level, nor with agreement rates with Sormunen and TREC judgments.…”

Section: Resultssupporting

confidence: 90%

“…In our work we investigate the feasibility of this approach in terms of individual and aggregated judgment quality. (Anderton et al 2013) look at mistakes by crowd workers performing relevance judgments, comparing to trained TREC assessors. They show that very short time spent to make the judgment leads to worse quality judgments and that document length has little impact on the time spent to make a judgment.…”

Section: Relevance Judgments and Crowdsourcingmentioning

confidence: 99%

See 1 more Smart Citation

Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge

Maddalena

Basaldella

Nart

et al. 2016

HCOMP

View full text Add to dashboard Cite

Crowdsourcing has become an alternative approach to collect relevance judgments at scale thanks to the availability of crowdsourcing platforms and quality control techniques that allow to obtain reliable results. Previous work has used crowdsourcing to ask multiple crowd workers to judge the relevance of a document with respect to a query and studied how to best aggregate multiple judgments of the same topic-document pair. This paper addresses an aspect that has been rather overlooked so far: we study how the time available to express a relevance judgment affects its quality. We also discuss the quality loss of making crowdsourced relevance judgments more efficient in terms of time taken to judge the relevance of a document. We use standard test collections to run a battery of experiments on the crowdsourcing platform CrowdFlower, studying how much time crowd workers need to judge the relevance of a document and at what is the effect of reducing the available time to judge on the overall quality of the judgments. Our extensive experiments compare judgments obtained under different types of time constraints with judgments obtained when no time constraints were put on the task. We measure judgment quality by different metrics of agreement with editorial judgments. Experimental results show that it is possible to reduce the cost of crowdsourced evaluation collection creation by reducing the time available to perform the judgments with no loss in quality. Most importantly, we observed that the introduction of limits on the time available to perform the judgments improves the overall judgment quality. Top judgment quality is obtained with 25-30 seconds to judge a topic-document pair.

show abstract

Section: Resultssupporting

confidence: 90%

Section: Relevance Judgments and Crowdsourcingmentioning

confidence: 99%

Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge

Maddalena

Basaldella

Nart

et al. 2016

HCOMP

View full text Add to dashboard Cite

show abstract

“…For example Alonso and Mizzaro [2012] compared relevance judgments gathered by the crowd vs. more experienced TREC assessors, finding comparable accuracy. Others have attempted to measure relevance in various ways, including graded judgments [McCreadie et al 2011], preferencebased judgments [Anderton et al 2013], and multidimensional ones [Zhang et al 2014]. Collecting relevance labels through crowdsourcing has also been used in practice, for example in the TREC blog track [McCreadie et al 2011], or in the judgment task of the TREC Crowdsourcing Track [Smucker et al 2014].…”

Section: Crowdsourcingmentioning

confidence: 99%

On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation

Maddalena

Mizzaro

Scholer

et al. 2017

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Magnitude estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of magnitude estimation for judging the relevance of documents for information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting over 50,000 magnitude estimation judgments using crowdsourcing. Our analysis shows that magnitude estimation judgments can be reliably collected using crowdsourcing, are competitive in terms of assessor cost, and are, on average, rank-aligned with ordinal judgments made by expert relevance assessors. We explore the application of magnitude estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and magnitude estimation relevance shows substantial variation; in particular, the top systems ranked using magnitude estimation and ordinal judgments differ substantially. Analysis of the magnitude estimation scores shows that this effect is due in part to varying perceptions of relevance: different users have different perceptions of the impact of relative differences in document relevance. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold.

show abstract

“…As Rogstadius et al [2011] show, the accuracy of outputs typically decreases with increasing task complexity. From a design perspective, it is further important to implement tasks in a way that limits cognitive complexity; for instance, comparing two objects is easier than identifying features of individual objects [Anderton et al 2013]. While the simplification of more complex tasks introduces longer completion times, it leads to higher quality; simpler tasks suit better if workers perform with interruptions [Cheng et al 2015b].…”

Section: Improve Task Designmentioning

confidence: 99%

Quality Control in Crowdsourcing

et al. 2018

View full text Add to dashboard Cite

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar -all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.

show abstract

An analysis of crowd workers mistakes for specific and complex relevance assessment task

Cited by 4 publications

References 4 publications

Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge

Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge

On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation

Quality Control in Crowdsourcing

Contact Info

Product

Resources

About