Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History

Abstract

Crowdsourcing more complex and creative tasks is seen as a desirable goal for both employers and workers, but these tasks traditionally require domain expertise. Employers can recruit only expert workers, but this approach does not scale well. Alternatively, employers can decompose complex tasks into simpler micro-tasks, but some domains, such as historical analysis, cannot be easily modularized in this way. A third approach is to train workers to learn the domain expertise. This approach offers clear benefits to workers, but is perceived as costly or infeasible for employers. In this paper, we explore the trade-offs between learning and productivity in training crowd workers to analyze historical documents. We compare CrowdSCIM, a novel approach that teaches historical thinking skills to crowd workers, with two crowd learning techniques from prior work and a baseline. Our evaluation (n=360) shows that CrowdSCIM allows workers to learn domain expertise while producing work of equal or higher quality versus other conditions, but efficiency is slightly lower.

Task Assignment Strategies for Crowd Worker Ability Improvement

“…In our new formulation, we seek to design holistic workflows that balance learning and productivity. In [56], a workflow called CrowdSCIM provides high learning gain but completion time is significantly longer (10 min. more) than other methods.…”

Making AI Machines Work for Humans in FoW

“…There has been a lot of work (e.g., [PW7]) on understanding the various factors affecting quality of work. Recent efforts such as [PW5] explore ways to improve worker's skill development through coaching while [PW6] discusses efficient mechanisms to teach crowd workers new skills. IC1 proposes mechanisms to capture skills (among other human factors) efficiently while IC3 talks about the challenges of upskilling.…”

Not Everyone Writes Good Examples but Good Examples Can Come from Anywhere

“…In addition to work on learnersourcing, there is an emerging body of work studying learning in crowdsourcing platforms. There are a number of studies that have looked into various ways of training crowdworkers, in addition to the ones mentioned above (Oleson et al 2011;Dow et al 2012;Singla et al 2014;Mamykina et al 2016;Streuer et al 2017;Wang, Hicks, and Luther 2018). Beyond simply training crowdworkers to do better on particular tasks, recent work has looked into understanding how crowdsourcing platforms can support learning as part of crowdwork and to foster the longer term development of worker skills (Krause et al 2016;Dontcheva et al 2014;Suzuki et al 2016;Jun, Arian, and Reinecke 2018).…”