AutoMan

Abstract

Humans can perform many tasks with ease that remain difficult or impossible for computers. Crowdsourcing platforms like Amazon's Mechanical Turk make it possible to harness human-based computational power at an unprecedented scale. However, their utility as a general-purpose computational platform remains limited. The lack of complete automation makes it difficult to orchestrate complex or interrelated tasks. Scheduling more human workers to reduce latency costs real money, and jobs must be monitored and rescheduled when workers fail to complete their tasks. Furthermore, it is often difficult to predict the length of time and payment that should be budgeted for a given task. Finally, the results of humanbased computations are not necessarily reliable, both because human skills and accuracy vary widely, and because workers have a financial incentive to minimize their effort. This paper introduces AUTOMAN, the first fully automatic crowdprogramming system. AUTOMAN integrates humanbased computations into a standard programming language as ordinary function calls, which can be intermixed freely with traditional functions. This abstraction lets AUTOMAN programmers focus on their programming logic. An AUTO-MAN program specifies a confidence level for the overall computation and a budget. The AUTOMAN runtime system then transparently manages all details necessary for scheduling, pricing, and quality control. AUTOMAN automatically schedules human tasks for each computation until it achieves the desired confidence level; monitors, reprices, and restarts human tasks as necessary; and maximizes parallelism across human workers while staying under budget.

A Review of Mobile Crowdsourcing Architectures and Challenges: Toward Crowd-Empowered Internet-of-Things

“…However, by asking a sufficient number of workers to perform the same task independently, we are able to gain the most common responses as the solution and expect a high quality of the correct (i.e., majority) answer. Barowy et al [53] optimize the majority voting approach by introducing an algorithm to estimate the confidence level of the responses that would be acceptable by the requester. Liu et al [54] studied improvements using a quality control mechanism relying on workers' past performances.…”

VoxPL

“…VOXPL modifications to AUTOMAN AUTOMAN is a modular crowdprogramming framework with integrated automatic scheduling, payment, and quality control for any given multiple-choice question and confidence level [4,5]. VOXPL generalizes and subsumes the AUTOMAN crowdprogramming framework, leveraging its automatic scheduling and pricing algorithms.…”

“…Later work takes a more statistical approach: many systems model latent variables like worker skill, worker speed, or task difficulty, or exploit correlations in the labels of similar inputs to predict accurate labels [9,35,29,8,10]. Several systems specifically address adversaries who may "game" the system by changing their behavior, treating the problem as a form of statistical noise rejection [4,31,27]. All of these approaches are designed with labeling in mind and do not provide quality guarantees for estimates.…”

PPLib

“…Automan [Barowy et al 2012], an automatic crowd programming system, fits this metaphor well and integrates human computation tasks as function calls in a standard programming language, thereby blurring the lines between CPUs and human processors. Among the main problems with programming the global brain is the fundamental difference between humans and computers in terms of motivational, error, and cognitive diversity [Bernstein et al 2012].…”