The Influence of Input Data Complexity on Crowdsourcing Quality

Tauchmann, Christopher; Daxenberger, Johannes; Mieskes, Margot

doi:10.1145/3379336.3381499

Cited by 4 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It allows annotators to progress through the annotation study similar to a game-by acquiring specific skills that are required to progress to the next level (Sweetser and Wyeth 2005). Although several works have shown the efficiency of progression in games with a purpose (Madge et al 2019;Kicikoglu et al 2020) and even in crowdsourcing (Tauchmann, Daxenberger, and Mieskes 2020), this does not necessarily benefit individual workers, as less-skilled workers are either filtered out or asked to "train" on additional instances. Moreover, implementing progression poses a substantial burden on researchers due to the inclusion of game-like elements (e.g., skills and levels), or at minimum, the separation of the data according to difficulty and, furthermore, a repeated evaluation and reassignment of workers.…”

Section: Related Workmentioning

confidence: 99%

“…Similar to educational approaches, we rely on estimating the "difficulty" of an instance to generate our curriculum (Taylor 1953;Beinborn, Zesch, and Gurevych 2014;Lee, Schwan, and Meyer 2019). In this work, we investigate an easy-instances-first strategy that has been shown to be a reasonable strategy in previous work (Tauchmann, Daxenberger, and Mieskes 2020); thereby sorting instances in ascending order according to their difficulty. Our C * is thus approximated by the ordered set S = {x 1 , .…”

Section: Approachmentioning

confidence: 99%

See 1 more Smart Citation

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lee

Klie

Gurevych

2022

Computational Linguistics

View full text Add to dashboard Cite

Annotation studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations; especially in citizen science or crowdsourcing scenarios where domain expertise is not required. To alleviate these issues, this work proposes annotation curricula, a novel approach to implicitly train annotators. The goal is to gradually introduce annotators into the task by ordering instances to be annotated according to a learning curriculum. To do so, this work formalizes annotation curricula for sentence- and paragraph-level annotation tasks, defines an ordering strategy, and identifies well-performing heuristics and interactively trained models on three existing English datasets. Finally, we provide a proof of concept for annotation curricula in a carefully designed user study with 40 voluntary participants who are asked to identify the most fitting misconception for English tweets about the Covid-19 pandemic. The results indicate that using a simple heuristic to order instances can already significantly reduce the total annotation time while preserving a high annotation quality. Annotation curricula thus can be a promising research direction to improve data collection. To facilitate future research—for instance, to adapt annotation curricula to specific tasks and expert annotation scenarios—all code and data from the user study consisting of 2,400 annotations is made available

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Approachmentioning

confidence: 99%

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lee

Klie

Gurevych

2022

Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Similar to educational approaches, we rely on estimating the "difficulty" of an instance to generate our curriculum (Taylor 1953;Beinborn, Zesch, and Gurevych 2014;Lee, Schwan, and Meyer 2019). In this work, we investigate an easy-instances-first strategy which has been shown to be a reasonable strategy in previous work (Tauchmann, Daxenberger, and Mieskes 2020); thereby sorting instances in ascending order according to their difficulty. Our C * is thus approximated by the ordered set…”

Section: Approachmentioning

confidence: 99%

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lee¹,

Klie²,

Gurevych³

2021

Preprint

View full text Add to dashboard Cite

Annotation studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations; especially in citizen science or crowd sourcing scenarios where domain expertise is not required and only annotation guidelines are provided. To alleviate these issues, this work proposes annotation curricula, a novel approach to implicitly train annotators. The goal is to gradually introduce annotators into the task by ordering instances that are annotated according to a learning curriculum. To do so, this work formalizes annotation curricula for sentence-and paragraph-level annotation tasks, defines an ordering strategy, and identifies well-performing heuristics and interactively trained models on three existing English datasets. Finally, a user study is conducted with 40 voluntary participants who are asked to identify the most fitting misconception for English tweets about the Covid-19 pandemic. The results show that using a simple heuristic to order instances can already significantly reduce the total annotation time while preserving a high annotation quality. Annotation curricula thus can provide a novel way to improve data collection. To facilitate future research, all code and data from the user study consisting of 2,400 annotations is made available. 1

show abstract

“…Moreover, as annotators are compensated not by the time they spend but rather by the number of annotated instances, they are compelled to work fast to maximize their monetary gain-which can negatively affect annotation quality (Drutsa et al, 2020) or even result in spamming (Hovy et al, 2013). It can also be difficult to find crowdworkers for the task at hand, for instance due to small worker pools for languages other than English (Pavlick et al, 2014;Frommherz and Zarcone, 2021) or because the task requires special qualifications (Tauchmann et al, 2020). Finally, the deployment of crowdsourcing remains ethically questionable due to undervalued payment (Fort et al, 2011;Cohen et al, 2016), privacy breaches, or even psychological harm on crowdworkers (Shmueli et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Lessons Learned from a Citizen Science Project for Natural Language Processing

Klie,

Lee,

Stowe

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Many Natural Language Processing (NLP) systems use annotated corpora for training and evaluation. However, labeled data is often costly to obtain and scaling annotation projects is difficult, which is why annotation tasks are often outsourced to paid crowdworkers. Citizen Science is an alternative to crowdsourcing that is relatively unexplored in the context of NLP. To investigate whether and how well Citizen Science can be applied in this setting, we conduct an exploratory study into engaging different groups of volunteers in Citizen Science for NLP by re-annotating parts of a pre-existing crowdsourced dataset. Our results show that this can yield high-quality annotations and attract motivated volunteers, but also requires considering factors such as scalability, participation over time, and legal and ethical issues. We summarize lessons learned in the form of guidelines and provide our code and data to aid future work on Citizen Science. 1

show abstract

The Influence of Input Data Complexity on Crowdsourcing Quality

Cited by 4 publications

References 7 publications

Annotation Curricula to Implicitly Train Non-Expert Annotators

Annotation Curricula to Implicitly Train Non-Expert Annotators

Annotation Curricula to Implicitly Train Non-Expert Annotators

Lessons Learned from a Citizen Science Project for Natural Language Processing

Contact Info

Product

Resources

About