Cheaper and Better: Selecting Good Workers for Crowdsourcing

Li, Hongwei; Liu, Qiang

doi:10.48550/arxiv.1502.00725

Cited by 4 publications

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Liu

2015

HCOMP

View full text Add to dashboard Cite

Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of top-quality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows.

show abstract

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Liu

2015

HCOMP

View full text Add to dashboard Cite

show abstract

Creating Task-Generic Features for Fake News Detection

Olivieri¹,

Shabani²,

Sokhn³

et al. 2019

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

Information spreads at a pace never seen before on online platforms, even when this information is fake. Fake news can have substantial impact, for instance when it concern politics and influences the results of legislations or elections. Finding a methodology to verify if some piece of news is true or false is hence essential. In this work, we propose a methodology to create task-generic features that are paired with textual features in order to detect fake news. Task-generic features are created by elaborating on metadata attached to answers from Google's search engine, and by using crowdsourcing for missing values. We experimentally validate our method on a dataset for fake news detection based on the PolitiFact website. Our results show an improvement in F1-Score of 3% over the state of the art, which is significant for a 6-class task.

show abstract

On the Cost Complexity of Crowdsourcing

Fang

Sun

Chen

et al. 2018

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Existing efforts mainly use empirical analysis to evaluate the effectiveness of crowdsourcing methods, which is often unreliable across experimental settings. Consequently, it is of great importance to study theoretical methods. This work, for the first time, defines the cost complexity of crowdsourcing, and presents two theorems to compute the cost complexity. Our theorems provide a general theoretical method to model the trade-off between costs and quality, which can be used to evaluate and design crowdsourcing algorithms, and characterize the complexity of crowdsourcing problems. Moreover, following our theorems, we prove a set of corollaries that can obtain existing theoretical results for special cases. We have verified our work theoretically and empirically.

show abstract

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Cited by 4 publications

References 8 publications

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Cheaper and Better: Selecting Good Workers for Crowdsourcing

Creating Task-Generic Features for Fake News Detection

On the Cost Complexity of Crowdsourcing

Contact Info

Product

Resources

About