Decision-Theoretic Control of Crowd-Sourced Workflows

Abstract

Crowd-sourcing is a recent framework in which human intelligence tasks are outsourced to a crowd of unknown people ("workers") as an open call (e.g., on Amazon's Mechanical Turk). Crowd-sourcing has become immensely popular with hoards of employers ("requesters"), who use it to solve a wide variety of jobs, such as dictation transcription, content screening, etc. In order to achieve quality results, requesters often subdivide a large task into a chain of bite-sized subtasks that are combined into a complex, iterative workflow in which workers check and improve each other's results. This paper raises an exciting question for AI — could an autonomous agent control these workflows without human intervention, yielding better results than today's state of the art, a fixed control program? We describe a planner, TurKontrol, that formulates workflow control as a decision-theoretic optimization problem, trading off the implicit quality of a solution artifact against the cost for workers to achieve it. We lay the mathematical framework to govern the various decisions at each point in a popular class of workflows. Based on our analysis we implement the workflow control algorithm and present experiments demonstrating that TurKontrol obtains much higher utilities than popular fixed policies.

Toward Automatic Bootstrapping of Online Communities Using Decision-theoretic Optimization

“…This gave us a list of 266,101 authors, along with the number of publications each has produced. Then, we randomly sampled 400 authors 11 and generated the corresponding number of synthetic papers for each author. After that, we constructed the citation graph using the rich-get-richer model [13].…”

“…For these communities, we define the bootstrapping process as a decision-theoretic optimization problem. Previous research has shown decisiontheoretic optimization is useful in similar social computing contexts such as crowdsourcing [11]. Applying the decisiontheoretic framework to model the bootstrapping problem allows us to estimate the utility of different operations and find a set of operations that are near optimal for the community.…”

“…SUP-PLE [17] renders interfaces using decision theory to optimize the ease of a user's expected behavior. Researchers have also used decision theory to control social interactions, especially for the specific application of optimizing crowdsourced workflows [11,23,33]. Our model of community bootstrapping, described in the next section, builds on this seminal work.…”

Ability Grouping of Crowd Workers via Reward Discrimination

“…Furthermore, we compared the obtained accuracy of our judgment rule with that of majority voting. @ In the field of machine-learning, several useful techniques to control qualities of crowdsourced tasks have been proposed and more elaborated machine-learning based methods for label aggregation exist (Dai, Mausam, and Weld 2010;Whitehill et al 2009). However, to estimate the ability of workers, they require each worker to do many tasks.…”

DataSift: An Expressive and Accurate Crowd-Powered Search Toolkit

“…By themselves, traditional information retrieval techniques are insufficient for our human-assisted retrieval task. On the other hand, existing crowd-powered systems, including Soylent (Bernstein et al 2010), Clowder (Dai, Mausam, and Weld 2010), Turkit (Little et al 2009), and CrowdPlan (Law and Zhang 2011) do not address the problem of improving information retrieval. Of these, perhaps the most similar is the work on CrowdPlan (Law and Zhang 2011), where human workers assist in breaking down a high level goal into smaller ones, and the operations performed by the human workers are similar to those proposed here.…”