Turkomatic

Kulkarni, Anand; Can, Matthew; Hartmann, Bjoern

doi:10.1145/1979742.1979865

Cited by 66 publications

(1 citation statement)

References 12 publications

(8 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their individual outputs are then compared and consolidated into a final solution (Shahaf and Horvitz, 2010). However, none of these steps is actually easy: some problems are less amenable to microtasking and need to be turned into bespoke microtask workflows Kulkarni et al, 2011;Kittur et al, 2011); the performance of the crowd varies across tasks (Mao et al, 2013;Redi and Povoa, 2014); and determining which answers are the most useful ones can be both complex and computationally expensive (Kittur et al, 2008;Snow et al, 2008;Vickrey et al, 2008;Demartini et al, 2012;Wiggins et al, 2011). It is on this last aspect, determining the correct answers, that we focus on in this paper.…”

Section: Introductionmentioning

confidence: 99%

Quality assessment in crowdsourced classification tasks

Simperl

Chapman

et al. 2019

IJCS

View full text Add to dashboard Cite

Purpose -Ensuring quality is one of the most significant challenges in microtask crowdsourcing tasks.Aggregation of the collected data from the crowd is one of the important steps to infer the correct answer, but the existing study seems to be limited to the single-step task. This study aims to look at multiple-step classification tasks and understand aggregation in such cases; hence, it is useful for assessing the classification quality.Design/methodology/approach -The authors present a model to capture the information of the workflow, questions and answers for both single-and multiple-question classification tasks. They propose an adapted approach on top of the classic approach so that the model can handle tasks with several multiplechoice questions in general instead of a specific domain or any specific hierarchical classifications. They evaluate their approach with three representative tasks from existing citizen science projects in which they have the gold standard created by experts.Findings -The results show that the approach can provide significant improvements to the overall classification accuracy. The authors' analysis also demonstrates that all algorithms can achieve higher accuracy for the volunteer-versus paid-generated data sets for the same task. Furthermore, the authors observed interesting patterns in the relationship between the performance of different algorithms and workflow-specific factors including the number of steps and the number of available options in each step.Originality/value -Due to the nature of crowdsourcing, aggregating the collected data is an important process to understand the quality of crowdsourcing results. Different inference algorithms have been studied for simple microtasks consisting of single questions with two or more answers. However, as classification tasks typically contain many questions, the proposed method can be applied to a wide range of tasks including both single-and multiple-question classification tasks.

show abstract

Section: Introductionmentioning

confidence: 99%

Quality assessment in crowdsourced classification tasks

Simperl

Chapman

et al. 2019

IJCS

View full text Add to dashboard Cite

show abstract

A crowdsourced “Who wants to be a millionaire?” player

Aydin

Yilmaz

Demirbaş

2017

Concurrency and Computation

View full text Add to dashboard Cite

Question answering is a fundamental problem for artificial intelligence research. State-of-the-art question answering systems, such as search engines, can answer well-formed factual questions. However, they fail on nonfactual and natural language queries. On the other hand, crowdsourcing leverages human intelligence to present solutions for problems, which are hard for computers. In the process of crowdsourcing, aggregating the responses from the crowd is a big challenge of itself. This work presents a model that integrates artificial intelligence with crowdsourcing to answer difficult natural language multiple choice questions. We build a crowdsourced mobile gaming experience for "Who wants to be a millionaire?" TV quiz show to test our methods. We use lightweight artificial intelligence models in aggregating the crowd responses. Our methods are able to answer even the hardest questions with very high accuracy. The experiment results suggest that building a super player for "Who wants to be a millionaire?" is feasible using our models. This study shows that facilitating crowdsourcing with artificial intelligence is an important tool for answering questions, which trouble state-of-the-art question answering systems.

show abstract

Decision Support Based on Human-Machine Collective Intelligence: Major Challenges

Smirnov

Ponomarev

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Turkomatic

Cited by 66 publications

References 12 publications

Quality assessment in crowdsourced classification tasks

Quality assessment in crowdsourced classification tasks

A crowdsourced “Who wants to be a millionaire?” player

Decision Support Based on Human-Machine Collective Intelligence: Major Challenges

Contact Info

Product

Resources

About