The Hidden Cost of Using Amazon Mechanical Turk for Research

Abstract

In this study, we investigate the attentiveness exhibited by participants sourced through Amazon Mechanical Turk (MTurk), thereby discovering a significant level of inattentiveness amongst the platform’s top crowd workers (those classified as ‘Master’, with an ‘Approval Rate’ of 98% or more, and a ‘Number of HITS approved’ value of 1,000 or more). A total of 564 individuals from the United States participated in our experiment. They were asked to read a vignette outlining one of four hypothetical technology products and then complete a related survey. Three forms of attention check (logic, honesty, and time) were used to assess attentiveness. Through this experiment we determined that a total of 126 (22.3%) participants failed at least one of the three forms of attention check, with most (94) failing the honesty check – followed by the logic check (31), and the time check (27). Thus, we established that significant levels of inattentiveness exist even among the most elite MTurk workers. The study concludes by reaffirming the need for multiple forms of carefully crafted attention checks, irrespective of whether participant quality is presumed to be high according to MTurk criteria such as ‘Master’, ‘Approval Rate’, and ‘Number of HITS approved’. Furthermore, we propose that researchers adjust their proposals to account for the effort and costs required to address participant inattentiveness.

Investigating the Split‐Attention Effect in Computer‐Based Assessment: Spatial Integration and Interactive Signaling Approaches

“…Research suggests that data collected from MTurk are psychometrically reliable (Buhrmester et al., 2011; Holden et al., 2013), MTurk participants are internally motivated (Buhrmester et al., 2011), and interactive experiments run through MTurk can be just as internally valid as those run in the lab or field experiments (Thomas & Clifford, 2017). On the other hand, some studies have raised concerns regarding MTurk participants’ inattentiveness (Chandler et al., 2014; Saravanos et al., 2021) and discrepancy with more nationally representative samples (Stone et al., 2019). In our study, the fact that participants’ overall performance in the prior knowledge test (0.46) and in the geometry test (0.51) was above the chance level (approximately 0.25) suggest that the current participants were reasonably engaged in solving the items.…”

Motives Linking Subclinical Psychopathy and Benign Masochism to Recreational Drug use

“…Given the sensitive nature of the topics of the present research, some participants may not have accurately reported on how often they use drugs, have dark personalities, and are prone to masochistic tendencies. Yet, the combination of online (and hence more anonymous) surveys, multiple IMCs, and plausibility checks should have strongly counteracted dishonest responding (Paolacci et al, 2010; Saravanos et al, 2021). Third, to achieve high statistical power, an online survey with a US-based community sample was carried out.…”

Extending the Technology Acceptance Model 3 to Incorporate the Phenomenon of Warm-Glow

“…We recruited our participants using the Amazon Mechanical Turk crowdsourcing platform, which has been used previously for such studies [40]. To identify an appropriate minimum size for our sample we looked to the "10 times rule method", which is "the most widely used minimum sample size estimation method in PLS-SEM" [41].…”