The viability of crowdsourcing for survey research

Abstract

Online contract labor portals (i.e., crowdsourcing) have recently emerged as attractive alternatives to university participant pools for the purposes of collecting survey data for behavioral research. However, prior research has not provided a thorough examination of crowdsourced data for organizational psychology research. We found that, as compared with a traditional university participant pool, crowdsourcing respondents were older, were more ethnically diverse, and had more work experience. Additionally, the reliability of the data from the crowdsourcing sample was as good as or better than the corresponding university sample. Moreover, measurement invariance generally held across these groups. We conclude that the use of these labor portals is an efficient and appropriate alternative to a university participant pool, despite small differences in personality and socially desirable responding across the samples. The risks and advantages of crowdsourcing are outlined, and an overview of practical and ethical guidelines is provided.

Amazon’s Mechanical Turk as a Viable Source for Organizational and Occupational Health Research

“…Behrend et al (2011) found that MTurk data was as good as or more reliable than university subject pools. Similarly, Buhrmester et al (2011) concluded that data obtained through MTurk was at least as reliable as those obtained through traditional methods, with high levels of average internal consistency of cross-sectional self-report data, as well as high levels of average test-retest reliabilities.…”

“…Similarly, Peer et al (2014) found that Turkers with high HIT approval ratings (95%) from requesters produce the highest quality data. Behrend et al (2011) found that MTurk data shows lower levels of social desirability responding than university subject pools. Meanwhile, Sprouse (2011) found that judgment experiments conducted through MTurk were generally indistinguishable from laboratory data.…”

“…Additionally, recent research by Shapiro et al (2013) demonstrated criterion-related validity of self-reported clinical symptoms, thus displaying the utility of MTurk as a resource for clinical psychology research on underrepresented psychiatric disorders and related problems. Despite these informative advances, a gap in the literature revolves around the labor market characteristics of Turkers, as this information is limited (e.g., Behrend et al 2011;Landers and Behrend 2015). The potential use of MTurk for organizational research, such as industrial and organizational psychology and occupational health science, could help address limitations of the existing literature.…”

“…Much of the research evaluating MTurk samples has compared multiple data sources (e.g., MTurk, social media, college students), concluding that MTurk samples are generally more demographically diverse than these alternatives (e.g., Behrend et al 2011;Christenson and Glick 2013). Additionally, recent research by Shapiro et al (2013) demonstrated criterion-related validity of self-reported clinical symptoms, thus displaying the utility of MTurk as a resource for clinical psychology research on underrepresented psychiatric disorders and related problems.…”

“…MTurk appears to provide a demographically diverse participant pool for behavioral and psychological research (Behrend et al 2011;Berinsky et al 2012;Mason and Suri 2012;Paolacci et al 2010). Much of the research evaluating MTurk samples has compared multiple data sources (e.g., MTurk, social media, college students), concluding that MTurk samples are generally more demographically diverse than these alternatives (e.g., Behrend et al 2011;Christenson and Glick 2013).…”