MTurk Research: Review and Recommendations

Abstract

The use of Amazon’s Mechanical Turk (MTurk) in management research has increased over 2,117% in recent years, from 6 papers in 2012 to 133 in 2019. Among scholars, though, there is a mixture of excitement about the practical and logistical benefits of using MTurk and skepticism about the validity of the data. Given that the practice is rapidly increasing but scholarly opinions diverge, the Journal of Management commissioned this review and consideration of best practices. We hope the recommendations provided here will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and related fields.

Do Privacy Concerns About Social Robots Affect Use Intentions? Evidence From an Experimental Vignette Study

“…The experimental vignette study was conducted in December 2018, in the form of a randomized survey with two conditions: high privacy risks and low privacy risks. We programmed the survey in Qualtrics and relied on MTurk for the participant recruitment, surveying respondents located in the United States (see Aguinis et al, 2020 for more information on MTurk as a data source). The average completion time was 8 min and participants were compensated with 1.25 US dollars, leading to an average hourly wage of 9.5 US dollars.…”

A little shot of humility: Intellectual humility predicts vaccination attitudes and intention to vaccinate against COVID‐19

“…However, our budget allowed us to collect data from additional participants for this project. We felt that a larger sample size would allow us some flexibility around our estimate and enable us to preemptively address anticipated data loss due to online data collection (Aguinis et al, 2020;Oppenheimer et al, 2009).…”

“…Participants were recruited from Amazon Mechanical Turk (MTurk), a popular source for participants in the social sciences (for review and recommendations, see Aguinis et al, 2020). A pilot version of the study was conducted to determine the average completion time.…”

“…For example, other practices to screen data include adding "attention check" items, although this approach may alter participants' responses in subsequent parts of the survey (Clifford & Jerit, 2015). Other approaches include screening participants' IP addresses (Kennedy et al, 2020) or response time (for a review of issues and recommendations for working with MTurk, see Aguinis et al, 2020). Future research may take efforts to include other data screening measures in addition to or in place of our approach to ensure data quality.…”

Leading the team, but feeling dissatisfied: Investigating informal leaders' energetic activation and work satisfaction and the supporting role of formal leadership

“…About 48 participants were removed from the sample because of their short (less than 5 min) response time. Another 62 participants were excluded as a result of failing one of three reading checks (see below), leaving a final sample of 210 (i.e., with a 34% attrition rate, which is common in studies using Mturk especially when careful data screening strategies are applied; see Aguinis et al, 2020). About half of respondents were males (53.4%), with an average work experience of 12.42 years (SD = 9.48).…”