Crowdsourced Knowledge Acquisition

Sabou, Marta; Scharl, Arno; Föls, Michael

doi:10.4018/ijswis.2013070102

Cited by 15 publications

(3 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar debate has been going on in the context of GWAPs, as designers are very restricted in assigning questions to difficulty levels without preprocessing them [43]. One option would be to try out a multi-step workflow (such as the hybrid workflow proposed by [39]) in which entity types that are empirically straightforward to annotate are solved by 'regular' workers, while miscellaneous and other problematic cases are only flagged and treated differently -be that by more experienced annotators, via a higher number of judgements [45], or otherwise.…”

Section: Discussionmentioning

confidence: 99%

An extended study of content and crowdsourcing-related performance factors in named entity annotation

Feyisetan

Simperl

Luczak‐Roesch

et al. 2018

View full text Add to dashboard Cite

Hybrid annotation techniques have emerged as a promising approach to carry out named entity recognition on noisy microposts. In this paper, we identify a set of content and crowdsourcing-related features (number and type of entities in a post, average length and sentiment of tweets, composition of skipped tweets, average time spent to complete the tasks, and interaction with the user interface) and analyse their impact on correct and incorrect human annotations. We then carried out further studies on the impact of extended annotation instructions and disambiguation guidelines on the factors listed above. This was all done using CrowdFlower and a simple, custom built gamified NER tool on three datasets from related literature and a fourth newly annotated corpus. Our findings show that crowd workers correctly annotate shorter tweets with fewer entities, while they skip (or wrongly annotate) longer tweets with more entities. Workers are also adept at recognising people and locations, while they have difficulties in identifying organisations and miscellaneous entities which they skip (or wrongly annotate). Finally, detailed guidelines do not necessarily lead to improved annotation quality. We expect these findings to lead to the design of more advanced NER pipelines, informing the way in which tweets are chosen to be outsourced to automatic tools, crowdsourced workers and nichesourced experts. Experimental results are published as JSON-LD for further use.

show abstract

Section: Discussionmentioning

confidence: 99%

An extended study of content and crowdsourcing-related performance factors in named entity annotation

Feyisetan

Simperl

Luczak‐Roesch

et al. 2018

View full text Add to dashboard Cite

show abstract

“…In the area of using Human Computation for Semantic Web research (HC4SW), there are a few trending topics both in the overall paper corpus we collected and in the special issue papers. For example, research on workflow design has considered workflows that combine different HC genre [1,39] as well as hybrid human-machine workflows [9,14]. The latter type of workflows dovetails with recent efforts to construct Human-in-the-Loop systems and still raises several open research issues as discussed in [9].…”

Section: Open Challenges and Future Workmentioning

confidence: 99%

Semantic Web and Human Computation: The status of an emerging field

Sabou

Aroyo

Bontcheva

et al. 2018

Self Cite

View full text Add to dashboard Cite

This editorial paper introduces a special issue that solicited papers at the intersection of Semantic Web and Human Computation research. Research in that inter-disciplinary space dates back a decade, and has been acknowledged as a research line of its own by a seminal research manifesto published in 2015. But where do we stand in 2018? How did this research line evolve during the last decade? How do the papers in this special issue align with the main lines of work of the community? In this editorial we inspect and reflect on the evolution of research at the intersection of Semantic Web and Human Computation. We use a methodology based on Systematic Mapping Studies to collect quantitative bibliographic data which we analyze through the lens of research topics envisioned by the research manifesto to characterize the evolution of research in this area, thus providing a context for introducing the papers of this special issue. We found evidences of a thriving research field; while steadily maturing, the field offers a number of open research opportunities for work where Semantic Web best practices and techniques are applied to support and improve the state-of-the-art in Human Computation, but also for work that exploits the strength of both areas to address scientifically and societally relevant issues.

show abstract

“…A way to handle this and collect cleaner data is to break the tasks down into smaller components (Sabou et al 2013). To make each task easier and faster, we will test a pipeline approach that will break the all-in-one task shown in Figs 2, 3, 4 into separate tasks performed by different Workers.…”

Section: Trainingmentioning

confidence: 99%

Crowdsourcing voice editing and quality assessment of data collected from the largest mobile phone-based research study of Parkinson disease

Klein¹

2016

RIO

View full text Add to dashboard Cite

Crowdsourced Knowledge Acquisition

Cited by 15 publications

References 44 publications

An extended study of content and crowdsourcing-related performance factors in named entity annotation

An extended study of content and crowdsourcing-related performance factors in named entity annotation

Semantic Web and Human Computation: The status of an emerging field

Crowdsourcing voice editing and quality assessment of data collected from the largest mobile phone-based research study of Parkinson disease

Contact Info

Product

Resources

About