Effects of Expertise Assessment on the Quality of Task Routing in Human Computation

Bry, François; Kneissl, Fabian; Krefeld, Thomas; Luecke, Stephan; Wieser, Christoph

doi:10.14236/ewic/sohuman2013.2

Cited by 15 publications

(15 citation statements)

References 19 publications

(17 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Task assignment, i.e., the intelligent matching of tasks with the most appropriate workers, is a fundamental challenge of crowdsourcing [20][21][22]. Although there have been several studies on conventional crowdsourcing task allocation, they cannot be directly applied to spatial crowdsourcing, because the location of the spatial task and that of the workers are vital for the result of the spatial task assignment.…”

Section: Related Workmentioning

confidence: 99%

TASC-MADM: Task Assignment in Spatial Crowdsourcing Based on Multiattribute Decision-Making

Chang

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

The methodology, formulating a reasonable task assignment to find the most suitable workers for a task and achieving the desired objectives, is the most fundamental challenge in spatial crowdsourcing. Many task assignment approaches have been proposed to improve the quality of crowdsourcing results and the number of task assignment and to limit the budget and the travel cost. However, these approaches have two shortcomings: (1) these approaches are commonly based on the attributes influencing the result of task assignment. However, different tasks may have different preferences for individual attributes; (2) the performance and efficiency of these approaches are expected to be improved further. To address the above issues, we proposed a task assignment approach in spatial crowdsourcing based on multiattribute decision-making (TASC-MADM), with the dual objectives of improving the performance as well as the efficiency. Specifically, the proposed approach jointly considers the attributes on the quality of the worker and the distance between the worker and the task, as well as the influence differences caused by the task’s attribute preference. Furthermore, it can be extended flexibly to scenarios with more attributes. We tested the proposed approach in a real-world dataset and a synthetic dataset. The proposed TASC-MADM approach was compared with the RB-TPSC and the Budget-TASC algorithm using the real dataset and the synthetic dataset; the TASC-MADM approach yields better performance than the other two algorithms in the task assignment rate and the CPU cost.

show abstract

Section: Related Workmentioning

confidence: 99%

TASC-MADM: Task Assignment in Spatial Crowdsourcing Based on Multiattribute Decision-Making

Chang

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

show abstract

“…As common practice [26], we adopt assessment tasks to quantify reliability: in each game, we compute player reliability as a function of the number of errors the player makes on those assessment tasks. The reliability value is 1 if the player makes no mistakes and decreases toward zero with increasing errors.…”

Section: Atomic Tasks and Truth Inferencementioning

confidence: 99%

A Framework to Build Games with a Purpose for Linked Data Refinement

Calegari¹,

Fiano²,

Celino³

2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

With the rise of linked data and knowledge graphs, the need becomes compelling to find suitable solutions to increase the coverage and correctness of datasets, to add missing knowledge and to identify and remove errors. Several approaches -mostly relying on machine learning and NLP techniques -have been proposed to address this refinement goal; they usually need a partial gold standard, i.e. some "ground truth" to train automatic models. Gold standards are manually constructed, either by involving domain experts or by adopting crowdsourcing and human computation solutions. In this paper, we present an open source software framework to build Games with a Purpose for linked data refinement, i.e. web applications to crowdsource partial ground truth, by motivating user participation through fun incentive. We detail the impact of this new resource by explaining the specific data linking "purposes" supported by the framework (creation, ranking and validation of links) and by defining the respective crowdsourcing tasks to achieve those goals. To show this resource's versatility, we describe a set of diverse applications that we built on top of it; to demonstrate its reusability and extensibility potential, we provide references to detailed documentation, including an entire tutorial which in a few hours guides new adopters to customize and adapt the framework to a new use case.

show abstract

“…In [13], test questions created from a generalized knowledge base are used to estimate the reliability of the new workers. Their result suggest that this approach performs better than using gold-standard tasks automated selection of knowledge base questions for quality control [12] used a hybrid approach of self-rating and gold-standard task for estimating the expertise of workers, however the self-assessment does not ensure high accuracy on the actual tasks.…”

Section: Related Workmentioning

confidence: 99%

CrowdED

Zaveri

Serrano

Desai

et al. 2018

Companion of the the Web Conference 2018 on the Web Conference 2018 - WWW '18

View full text Add to dashboard Cite

People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.• The final author version and the galley proof are versions of the publication after peer review.• The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User Agreement:

show abstract

Effects of Expertise Assessment on the Quality of Task Routing in Human Computation

Cited by 15 publications

References 19 publications

TASC-MADM: Task Assignment in Spatial Crowdsourcing Based on Multiattribute Decision-Making

TASC-MADM: Task Assignment in Spatial Crowdsourcing Based on Multiattribute Decision-Making

A Framework to Build Games with a Purpose for Linked Data Refinement

CrowdED

Contact Info

Product

Resources

About