Two Tools are Better Than One

Song, Jean; Fok, Raymond; Lundgard, Alan; Yang, Fan; Kim, Juho; Lasecki, Walter S.

doi:10.1145/3172944.3172948

Cited by 10 publications

(1 citation statement)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Machine learning approaches have often assumed annotator labels are fixed and focused on modeling annotator reliabilities for better aggregation [2,3,14,18]. In contrast, human-centered approaches have sought to improve the quality of annotator work by improving the task design [1,7,15,19], or recording and visualizing worker behavior [12,13].…”

mentioning

confidence: 99%

Explainable modeling of annotations in crowdsourcing

Nguyen

Lease

Wallace

2019

Proceedings of the 24th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

Aggregation models for improving the quality of annotations collected via crowdsourcing have been widely studied, but far less has been done to explain why annotators make the mistakes that they do. To this end, we propose a joint aggregation and worker clustering model that detects patterns underlying crowd worker labels to characterize varieties of labeling errors. We evaluate our approach on a Named Entity Recognition dataset labeled by Mechanical Turk workers in both a retrospective experiment and a small human study. The former shows that our joint model improves the quality of clusters vs. aggregation followed by clustering. Results of the latter suggest that clusters aid human sense-making in interpreting worker labels and predicting worker mistakes. By enabling better explanation of annotator mistakes, our model creates a new opportunity to help Requesters improve task instructions and to help crowd annotators learn from their mistakes. Source code, data, and supplementary material is shared online.

show abstract

mentioning

confidence: 99%