Skill Ontology-Based Model for Quality Assurance in Crowdsourcing

Maarry, Kinda El; Balke, Wolf‐Tilo; Cho, Hyunsouk; Hwang, Seung Won; Baba, Yukino

doi:10.1007/978-3-662-43984-5_29

Cited by 19 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many solutions exist to resolve the low-quality contributions that come either from deliberate scamming or poor performance (Ipeirotis et al, 2010;Dow et al, 2012;Ipeirotis et al, 2014;Gadiraju et al, 2015). The most commonly adopted technique for quality assurance relies on a set of questions with known correct answers, often referred to as gold standard test questions, which are randomly injected into the workers' sessions in order to measure their performance (Checco et al, 2018;Sun and Dance, 2012;El Maarry et al, 2014). In addition, since the original Dawid-Sekene (Dawid and Skene, 1979) crowdsourcing model based on maximum likelihood estimation, several improvements have been proposed: the Fast Dawid-Skene model for improved speed of convergence (Sinha et al, 2018), the effort to extend to multiclass labeling from binary labeling (Li and Yu, 2014), and the Minmax Entropy method (Zhou et al, 2012) for accuracy improvements.…”

Section: Quality Assurance In Crowdsourcingmentioning

confidence: 99%

Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

Sun¹,

Kotek²,

Klein³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing the resulting error of the labeling process by as much as 20-30% in comparison to other common labeling strategies. Importantly, this new approach to the labeling process, which we name Dynamic Automatic Conflict Resolution (DACR), does not require a ground truth dataset and is instead based on inter-project annotation inconsistencies. This makes DACR not only more accurate but also available to a broad range of labeling tasks. In what follows we present results from a text classification task performed at scale for a commercial personal assistant, and evaluate the inherent ambiguity uncovered by this annotation strategy as compared to other common labeling strategies.

show abstract

Section: Quality Assurance In Crowdsourcingmentioning

confidence: 99%

Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

Sun¹,

Kotek²,

Klein³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…The reputation of workers is also commonly used as a metric for determining the quality of their responses, such as in [4], [5], and [30]. The reputation system reflects workers' task performance history.…”

Section: ) Worker Reputationmentioning

confidence: 99%

“…We reuse and extend some parts of the enterprise crowdsourcing ontology and adapt it for our purposes. In addition, a skill ontology-based model for quality assurance was suggested in [30]. While this model was used to identify and match the best worker to a given task, our approach aims to identify and match the best-suited quality control mechanism to a given task.…”

Section: Relevant Workmentioning

confidence: 99%

A Dynamic Selection Approach for Quality Control Mechanisms in Crowdsourcing

Alabduljabbar

Al-Dossari

2019

IEEE Access

View full text Add to dashboard Cite

Crowdsourcing has emerged as a new model for leveraging human knowledge and intelligence toward accomplishing tasks that are difficult to fulfill effectively with machines alone. However, owing to its open nature, quality control is a big challenge. Current crowdsourcing systems use one or two standard mechanisms for evaluation and quality control of a task, regardless of its type. In this paper, we propose a dynamic approach that exploits task-quality ontology to select the most suitable quality control mechanism (QCM) for a given task based on its type. The proposed approach has been enriched by a reputation engine that collects requesters' feedback on the performance of QCMs. Accordingly, QCMs and tasks were automatically matched using the underlying categorization structure of tasks on one side and the reputation scores of QCMs on the other side. Our experiments establish that our proposed dynamic approach yields better results compared to existing approaches.

show abstract

“…For example, ontologies of tasks allow improved participant selection in mobile crowdsourcing settings [47] and semantic descriptions of workflows facilitate the crowdsourcing of a constitution [27]. Another line of work focuses on describing the workers, their CVs and skills [6,28,41].…”

Section: Semantic Web For Human Computationmentioning

confidence: 99%

Semantic Web and Human Computation: The status of an emerging field

Sabou

Aroyo

Bontcheva

et al. 2018

View full text Add to dashboard Cite

This editorial paper introduces a special issue that solicited papers at the intersection of Semantic Web and Human Computation research. Research in that inter-disciplinary space dates back a decade, and has been acknowledged as a research line of its own by a seminal research manifesto published in 2015. But where do we stand in 2018? How did this research line evolve during the last decade? How do the papers in this special issue align with the main lines of work of the community? In this editorial we inspect and reflect on the evolution of research at the intersection of Semantic Web and Human Computation. We use a methodology based on Systematic Mapping Studies to collect quantitative bibliographic data which we analyze through the lens of research topics envisioned by the research manifesto to characterize the evolution of research in this area, thus providing a context for introducing the papers of this special issue. We found evidences of a thriving research field; while steadily maturing, the field offers a number of open research opportunities for work where Semantic Web best practices and techniques are applied to support and improve the state-of-the-art in Human Computation, but also for work that exploits the strength of both areas to address scientifically and societally relevant issues.

show abstract

Skill Ontology-Based Model for Quality Assurance in Crowdsourcing

Cited by 19 publications

References 18 publications

Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

A Dynamic Selection Approach for Quality Control Mechanisms in Crowdsourcing

Semantic Web and Human Computation: The status of an emerging field

Contact Info

Product

Resources

About