How serious is the ‘carelessness’ problem on Mechanical Turk?

Aruguete, Mara S.; Huynh, Ho Phi; Browne, Blaine L.; Jurs, Bethany S.; Flint, Emilia; McCutcheon, Lynn E.

doi:10.1080/13645579.2018.1563966

Cited by 67 publications

(41 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although researchers often do not report HIT requirements, it is important to note that the current research used HIT qualifications that were less stringent than previous recommendations (Peer, Vosgerau, & Acquisti, 2014), which could have impacted the generalizability of our results. However, the percentage of participants failing validity indicators in Waves 3 and 4 is similar to informal reports and recent publications (Aruguete et al, 2019; Courrégé, Skeel, Feder, & Boress, 2019; Dreyfuss, 2018). Moreover, the results were nearly identical in an additional Wave (4a) of data collected using HIT requirements commonly reported in the literature.…”

Section: Discussionsupporting

confidence: 88%

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

Chmielewski

Kucker

2019

Social Psychological and Personality Science

715

470

View full text Add to dashboard Cite

Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.

show abstract

Section: Discussionsupporting

confidence: 88%

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

Chmielewski

Kucker

2019

Social Psychological and Personality Science

715

470

View full text Add to dashboard Cite

show abstract

“…Recently, such pools have been under question for the reliability of the data collected from MTurk workers (e.g. Aruguete et al, 2019;Rouse, 2015). Additionally, researchers have highlighted the concern of professional participants, or "super workers," and bots in these sample pools (Chandler, Mueller, & Paolacci, 2014;Chmielewski & Kucker, 2019;Dennis et al, 2019;Fort, Adda, & Cohen, 2011;Stewart et al, 2015).…”

Section: Recruit From a Reliable Sourcementioning

confidence: 99%

Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks

Storozuk¹,

Ashley²,

Delage³

et al. 2020

TQMP

117

View full text Add to dashboard Cite

“…Then, we used a questionnaire for data reliability validation and weights for consistency comparison, except in the case of the HITAR. However, we needed to consider the casual observation characteristics of MTurk workers, because in this experiment, there was no HITAR constraint [100]. Therefore, we added five questions that induced the participants to enter as many words as possible in the LSA index of the DRW [65].…”

Section: Observation Of Future Reward Distribution Effectmentioning

confidence: 99%

Reliable Data Collection in Participatory Trials to Assess Digital Healthcare Applications

Park

Kim

et al. 2020

IEEE Access

View full text Add to dashboard Cite

The number of digital healthcare mobile applications in the market is exponentially increasing owing to the development of mobile networks and widespread usage of smartphones. However, only few of these applications have been adequately validated. Like many mobile applications, in general, the use of healthcare applications is considered safe; thus, developers and end users can easily exchange them in the marketplace. However, existing platforms are unsuitable for collecting reliable data for evaluating the effectiveness of the applications. Moreover, these platforms reflect only the perspectives of developers and experts, and not of end users. For instance, typical clinical trial data collection methods are not appropriate for participant-driven assessment of healthcare applications because of their complexity and high cost. Thus, we identified the need for a participant-driven data collection platform for end users that is interpretable, systematic, and sustainable, as a first step to validate the effectiveness of the applications. To collect reliable data in the participatory trial format, we defined distinct stages for data preparation, storage, and sharing. The interpretable data preparation consists of a protocol database system and semantic feature retrieval method that allow a person without professional knowledge to create a protocol. The systematic data storage stage includes calculation of the collected data reliability weight. For sustainable data collection, we integrated a weight method and a future reward distribution function. We validated the methods through statistical tests involving 718 human participants. The results of a validation experiment demonstrate that the compared methods differ significantly and prove that the choice of an appropriate method is essential for reliable data collection, to facilitate effectiveness validation of digital healthcare applications. Furthermore, we created a Web-based system for our pilot platform to collect reliable data in an integrated pipeline. We compared the platform features using existing clinical and pragmatic trial data collection platforms. INDEX TERMS Digital health, digital healthcare app data collection platform, crowdsourcing, participatory trial, biomedical informatics.

show abstract

How serious is the ‘carelessness’ problem on Mechanical Turk?

Cited by 67 publications

References 16 publications

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks

Reliable Data Collection in Participatory Trials to Assess Digital Healthcare Applications

Contact Info

Product

Resources

About