Explainable modeling of annotations in crowdsourcing

Abstract

Aggregation models for improving the quality of annotations collected via crowdsourcing have been widely studied, but far less has been done to explain why annotators make the mistakes that they do. To this end, we propose a joint aggregation and worker clustering model that detects patterns underlying crowd worker labels to characterize varieties of labeling errors. We evaluate our approach on a Named Entity Recognition dataset labeled by Mechanical Turk workers in both a retrospective experiment and a small human study. The former shows that our joint model improves the quality of clusters vs. aggregation followed by clustering. Results of the latter suggest that clusters aid human sense-making in interpreting worker labels and predicting worker mistakes. By enabling better explanation of annotator mistakes, our model creates a new opportunity to help Requesters improve task instructions and to help crowd annotators learn from their mistakes. Source code, data, and supplementary material is shared online.

Design and Evaluation of Crowd-sourcing Platforms Based on Users’ Confidence Judgments

“…• Algorithms based on machine learning and pattern recognition: these algorithms are one of the main answer aggregation algorithms in the field of annotation. For example, 21 found the final answer through clustering. Also, in 22 , it detects spam text messages using SVM and Bayesian filters.…”

“…Therefore, except for people who are very interested in competition, other people will not have the motivation to participate 25 . Another method is to pay a fixed amount to all the users 21,26 , which reduces the number of calculations to find the payment amount and gives everyone an incentive to participate in the task. Also, this method does not incentivize people to cheat for more money.…”