Crowdsourcing in Computing Education Research: Case Amazon MTurk

Abstract

Crowdsourcing platforms such as Amazon MTurk provide access to a human workforce that can be given tasks to complete online for a fee. In this article, we review studies in computing education research (CER) that rely on crowdsourcing; we also describe our own experiences of using Amazon MTurk for a CER study. We discuss challenges in recruiting workers with specific backgrounds-such as no programming experience-and considerations in filtering out unreliable research participants. Combining recommendations from the literature with the lessons that we learned whilst conducting our study, we synthesize advice for researchers in CER who are considering crowdsourcing. In our case study, we did not find widespread foul play by crowdsourced workers and, overall, our experiences and the literature suggest that crowdsourced CER is feasible. It is, however, uncertain to what extent crowdsourced data can produce answers that apply to specific educational contexts. More research is needed before definitive conclusions can be drawn about the validity and generalizability of crowdsourced CER. CCS CONCEPTS• Social and professional topics → Computing education.

Measuring College Student Attitudes Toward COVID-19 Vaccinations

“…MTurk has been found to be a unique and robust source of human intelligence services, including survey completion in educational contexts (Follmer et al, 2017). Several recent studies in education focused on financial aid jargon (Taylor & Bicak, 2019) and computer science education (Hellas et al, 2020) have used MTurk to answer research questions that require a large, nationally representative dataset, akin to the study at hand related to postsecondary student attitudes toward COVID-19 vaccinations.…”

Measuring College Student Attitudes Toward COVID-19 Vaccinations

“…MTurk has been found to be a unique and robust source of human intelligence services, including survey completion in educational contexts (Follmer et al, 2017). Several recent studies in education focused on financial aid jargon (Authors, 2019) and computer science education (Hellas et al, 2020) have used MTurk to answer research questions that require a large, nationally representative dataset, akin to the study at hand related to postsecondary student attitudes toward COVID-19 vaccinations. Overall, this study's survey sample was entirely recruited from the MTurk system, representing both a limitation and a strength of this study, which will be addressed later.…”

Lessons Learned From Four Computing Education Crowdsourcing Systems

“…Although paid crowdsourcing has been used within computing education research (see e.g. [31]- [36]), data quality has not received significant attention [33]. The concerns about data quality apply equally to learnersourcing contexts where students are not paid; this is a common instructor concern that we look to address in the lessons we synthesize in the current work.…”