The anatomy of a large-scale human computation engine

Kochhar, Shailesh; Mazzocchi, Stefano; Paritosh, Praveen

doi:10.1145/1837885.1837890

Cited by 55 publications

(33 citation statements)

References 9 publications

(10 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, a crowd of workers whose work is high-quality can be selected by qualifying tests or previous performance. These good workers are then kept for future tasks by paying an explicit bonus [76] or by forming a worker community around the system [37].…”

Section: Crowd Availabilitymentioning

confidence: 99%

See 2 more Smart Citations

Crowdsourced Monolingual Translation

Chang

Resnik

Bederson

2014

ACM Trans. Comput.-Hum. Interact.

View full text Add to dashboard Cite

An enormous potential exists for solving certain classes of computational problems through rich collaboration among crowds of humans supported by computers. Solutions to these problems used to involve human professionals who are expensive to hire or difficult to find. Despite significant advances, fully automatic systems still have much room for improvement. Recent research has involved recruiting large crowds of skilled humans A general protocol to handle crowdsourced monolingual translation is introduced along with three systems that implement the protocol. The MonoTrans system initially established the feasibility of the protocol. Then, MonoTrans2 enabled lab experiments with a second implementation of the protocol. MonoTrans2 was also applied to a an emergency-response scenario in a developing country (Haiti). The MonoTrans Widgets system was deployed to a large crowd of casual web users with a third implementation of the protocol. These systems were studied in various settings, and were found to supply improvement in quality over both machine translation and monolingual post-editing.

show abstract

Section: Crowd Availabilitymentioning

confidence: 99%

“…37 Victoria Elena cree que el ratón quiere el diente para construir una escalera muy larga para llegar a la luna, que está hecha de queso.…”

unclassified

See 1 more Smart Citation

Crowdsourced Monolingual Translation

Chang

Resnik

Bederson

2014

ACM Trans. Comput.-Hum. Interact.

View full text Add to dashboard Cite

show abstract

“…Approaches include task design, especially getting the work done iteratively (e.g. find, fix, verify for correcting documents [8]) and/or rating workers through reputation schemes. Whilst it is possible to simply weed out bad work/workers, a more promising approach is to distinguish between scammers and genuine workers and to enable on the job learning so that genuine workers can improve.…”

Section: Crowdsourcing Literaturementioning

confidence: 99%

Relationship-Based Business Process Crowdsourcing?

O’Neill

Martín

2013

Human-Computer Interaction – INTERACT 2013

View full text Add to dashboard Cite

Abstract. New technologies do not always benefit the worker, especially when harnessed by organisations seeking ever cheaper labour. Crowdsourcing is a technology-enabled way of working which offers the potential to bring work to far flung communities. However, it is something of a double-edged sword and there are many socio-technical and ethical challenges. In the micro-task market crowdsourcing platforms tend to be designed largely for the advantage of the organisation requesting work, rather than the worker. This paper contributes to research calling to redress this balance [2,6]. It describes the findings of an ethnographic study of an outsourced business process -healthcare form digitization -as performed by workers in-office (India) and @Home (USA). It reveals the complexities of the relationships between worker and organisation and argues that designing some aspects of these relationships into crowdsourcing platforms and applications is as beneficial for the organisation as it is for the worker.

show abstract

“…In [20] the authors thus reached the conclusion that an hourly payment was better (with some verification and time justification procedures).…”

mentioning

confidence: 99%

Crowdsourcing for Language Resource Development: Criticisms About Amazon Mechanical Turk Overpowering Use

Fort¹,

Adda²,

Sagot³

et al. 2014

Human Language Technology Challenges for Computer Science and Linguistics

View full text Add to dashboard Cite

Abstract. This article is a position paper about Amazon Mechanical Turk, the use of which has been steadily growing in language processing in the past few years. According to the mainstream opinion expressed in articles of the domain, this type of on-line working platforms allows to develop quickly all sorts of quality language resources, at a very low price, by people doing that as a hobby. We shall demonstrate here that the situation is far from being that ideal. Our goal here is manifold: 1-to inform researchers, so that they can make their own choices, 2-to develop alternatives with the help of funding agencies and scientific associations, 3-to propose practical and organizational solutions in order to improve language resources development, while limiting the risks of ethical and legal issues without letting go price or quality, 4-to introduce an Ethics and Big Data Charter for the documentation of language resources.

show abstract

The anatomy of a large-scale human computation engine

Cited by 55 publications

References 9 publications

Crowdsourced Monolingual Translation

Crowdsourced Monolingual Translation

Relationship-Based Business Process Crowdsourcing?

Crowdsourcing for Language Resource Development: Criticisms About Amazon Mechanical Turk Overpowering Use

Contact Info

Product

Resources

About