Collaboratively crowdsourcing workflows with turkomatic

Abstract

Preparing complex jobs for crowdsourcing marketplaces requires careful attention to workflow design, the process of decomposing jobs into multiple tasks, which are solved by multiple workers. Can the crowd help design such workflows? This paper presents Turkomatic, a tool that recruits crowd workers to aid requesters in planning and solving complex jobs. While workers decompose and solve tasks, requesters can view the status of worker-designed workflows in real time; intervene to change tasks and solutions; and request new solutions to subtasks from the crowd. These features lower the threshold for crowd employers to request complex work. During two evaluations, we found that allowing the crowd to plan without requester supervision is partially successful, but that requester intervention during workflow planning and execution improves quality substantially. We argue that Turkomatic's collaborative approach can be more successful than the conventional workflow design process and discuss implications for the design of collaborative crowd planning systems.

Crowdsourcing Processes: A Survey of Approaches and Opportunities

“…• Anand Kulkarni and his colleagues 6 crowdsource article writing (an article about the attractions of New York City) that involves tasks like structuring an article, writing narrative, splitting content into sections, adding pictures, iterating over content, and coordinating workers that write, correct, or structure text.…”

Question-Answer Cards for an Inclusive Micro-tasking Framework for the Elderly

“…Kulkarni et al proposed a framework to decompose complex tasks into micro-tasks [ 16]. They showed that even the work of decomposing the complex tasks into simple tasks could be crowd-sourced.…”

Quality Control in Crowdsourcing Systems: Issues and Directions

“…14,15 Designing workf lows for complex tasks greatly affects outcome quality. 1,2,5 For instance, one study demonstrated that designing a poor outline for an essay that the crowd will write can result in a low-quality essay.…”