A Survey of Human Computation Systems

Yuen, MF; Chen, Ling-Jyh; King, Irwin

doi:10.1109/cse.2009.395

Cited by 80 publications

(61 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Several attempts have been made recently to bring order to the rapidly developing field of collaborative creation on the Internet [62,46,80]. Wikipedia showed that allowing users free reign of encyclopaedic knowledge not only empowers mass participation but also that the resulting creation is of a very high quality.…”

Section: Collaborative Creation and Collective Intelligencementioning

confidence: 99%

Using Games to Create Language Resources: Successes and Limitations of the Approach

Chamberlain

Fort

Kruschwitz

et al. 2013

The People’s Web Meets NLP

View full text Add to dashboard Cite

One of the more novel approaches to collaboratively creating language resources in recent years is to use online games to collect and validate data. The most significant challenges collaborative systems face are how to train users with the necessary expertise and how to encourage participation on a scale required to produce high quality data comparable with data produced by "traditional" experts. In this chapter we provide a brief overview of collaborative creation and the different approaches that have been used to create language resources, before analysing games used for this purpose. We discuss some key issues in using a gaming approach, including task design, player motivation and data quality, and compare the costs of each approach in terms of development, distribution and ongoing administration. In conclusion, we summarise the benefits and limitations of using a gaming approach to resource creation and suggest key considerations for evaluating its utility in different research scenarios.

show abstract

Section: Collaborative Creation and Collective Intelligencementioning

confidence: 99%

Using Games to Create Language Resources: Successes and Limitations of the Approach

Chamberlain

Fort

Kruschwitz

et al. 2013

The People’s Web Meets NLP

View full text Add to dashboard Cite

show abstract

“…In the area of human computation, Yuen et al [82] identify three broad categories based on the relative maturity of the system. Initiatory systems are the earliest examples of human computation and were generally used to collect commonsense knowledge.…”

Section: Previous Frameworkmentioning

confidence: 99%

“…With the advent of online marketplaces providing an on-demand workforce for microtasks, we have seen an explosion of work utilizing human processing power to approach problems such as image labeling [21,33,73,74], digitizing text [77], annotating audio clips [44,49], and folding proteins [20]. For surveys of research in this area, please see [55,82]. While much of the work in this has shown promise in harnessing human computational power, there is a temptation to use human workers as an easy out.…”

Section: Human Computationmentioning

confidence: 99%

An Affordance-Based Framework for Human Computation and Human-Computer Collaboration

Crouser

Chang

2012

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

Abstract-VisualAnalytics is "the science of analytical reasoning facilitated by visual interactive interfaces" [70]. The goal of this field is to develop tools and methodologies for approaching problems whose size and complexity render them intractable without the close coupling of both human and machine analysis. Researchers have explored this coupling in many venues: VAST, Vis, InfoVis, CHI, KDD, IUI, and more. While there have been myriad promising examples of human-computer collaboration, there exists no common language for comparing systems or describing the benefits afforded by designing for such collaboration. We argue that this area would benefit significantly from consensus about the design attributes that define and distinguish existing techniques. In this work, we have reviewed 1,271 papers from many of the top-ranking conferences in visual analytics, human-computer interaction, and visualization. From these, we have identified 49 papers that are representative of the study of human-computer collaborative problem-solving, and provide a thorough overview of the current state-of-the-art. Our analysis has uncovered key patterns of design hinging on humanand machine-intelligence affordances, and also indicates unexplored avenues in the study of this area. The results of this analysis provide a common framework for understanding these seemingly disparate branches of inquiry, which we hope will motivate future work in the field.

show abstract

“…In parallel, Yuen et al (2009) also surveyed crowdsourcing applications, categorizing them into six classes: initiatory human computation, distributed human computation, social game-based human computation with volunteers, paid engineers and online players. Their survey analyzes crowdsourcing from a social gaming perspective, differentiating the classes based on game structure and mechanisms.…”

Section: Related Survey Work On Crowdsourcingmentioning

confidence: 99%

Perspectives on crowdsourcing annotations for natural language processing

Wang

Hoang

Kan

2012

Lang Resources & Evaluation

View full text Add to dashboard Cite

Crowdsourcing has emerged as a new method for obtaining annotations for training models for machine learning. While many variants of this process exist, they largely differ in their method of motivating subjects to contribute and the scale of their applications. To date, however, there has yet to be a study that helps the practitioner to decide what form an annotation application should take to best reach its objectives within the constraints of a project. We first provide a faceted analysis of existing crowdsourcing annotation applications. We then use our analysis to discuss our recommendations on how practitioners can take advantage of crowdsourcing and discuss our view on potential opportunities in this area.

show abstract

A Survey of Human Computation Systems

Cited by 80 publications

References 41 publications

Using Games to Create Language Resources: Successes and Limitations of the Approach

Using Games to Create Language Resources: Successes and Limitations of the Approach

An Affordance-Based Framework for Human Computation and Human-Computer Collaboration

Perspectives on crowdsourcing annotations for natural language processing

Contact Info

Product

Resources

About