Increasing cheat robustness of crowdsourcing tasks

Abstract

Crowdsourcing successfully strives to become a widely used means of collecting large-scale scientific corpora. Many research fields, including Information Retrieval, rely on this novel way of data acquisition. However, it seems to be undermined by a significant share of workers that are primarily interested in producing quick generic answers rather than correct ones in order to optimise their time-efficiency and, in turn, earn more money. Recently, we have seen numerous sophisticated schemes of identifying such workers. Those, however, often require additional resources or introduce artificial limitations to the task. In this work, we take a different approach by investigating means of a priori making crowdsourced tasks more resistant against cheaters.

A Hybrid Machine-Crowd Approach to Photo Retrieval Result Diversification

“…However crowd-sourcing is not a perfect system, issues such as quality control and reliability of results need further investigation. Automatic quality control methods, as the gold units used in this study, can be a good option since data cleaning is a time-consuming and tedious task [14]. However, they do not always ensure high quality data; since answers can only be rejected at runtime, they may attract more spammers, malicious, and sloppy workers [12] [14].…”

Quality Control Methods in Crowd Computing: Literature Review

“…В работах [68,69] делается попытка выделения причин получения некачественных результатов в крауд-вычислениях. Среди основных факторов, влияющих на качество, названы компетентность, заинтересованность в выполнении заданий (мотивация), ясность представления заданий, наличие или отсутствие «злого умысла» (целенаправленного подрыва работы системы).…”

Crowdsourcing Analysis of Twitter Data on Climate Change: Paid Workers vs. Volunteers

“…A commonly used solution is to employ a worker reputation system with assigning tasks to workers with approval ratings above a certain pre-set level [33]. Another set of methods of identification and expulsion of the unethical workers is based on a set of indices measuring (1) agreement with the expert "golden standard" data; (2) agreement with the other workers; (3) agreement with the attention check questions and (4) an amount of effort estimated from the task completion time [34]. The "golden standard" is a subset of data that is processed by experts in the field; an important condition is that a lay person should be able to process this data easily and unambitiously.…”

“…The attention check and language comprehension questions are verifiable questions [29] that do not require factual knowledge [36]; the results obtained from the workers failing to answer the attention questions correctly should be discarded. Finally, the average time to complete a single task is used to identify low-quality workers presumably spending a lesser amount of time per task [34].…”

“…For groundtruthing purposes, the "Gold Standard" tweets were selected based on [34]. For this, 579 tweets were screened by the first author, who was a climate scientist, and tweets that most easily and transparently could be classified into one of the classification categories were selected (103 in total).…”