Common Concerns with MTurk as a Participant Pool: Evidence and Solutions

Hauser, David; Paolacci, Gabriele; Chandler, Jesse

doi:10.31234/osf.io/uq45c

Cited by 160 publications

(192 citation statements)

References 93 publications

Supporting

Mentioning

187

Contrasting

Order By: Relevance

“…This does not preclude the possibility that workers have participated in other experimental studies that were similar to the current project (Chandler, Mueller, & Paolacci, ). Hauser, Paolacci, and Chandler () suggested that nonnaiveté among MTurk workers may be an issue when the HIT involves popular tests (e.g., the Cognitive Reflection Test) that, with experience, workers may learn how to execute. Meyer, Zhou, and Fredrick (), however, suggested that exposure to CRT tasks does not necessarily improve test scores.…”

mentioning

confidence: 99%

Fight or flight: Integral emotions and violent intentions

Barnum¹,

Solomon

2019

Criminology

View full text Add to dashboard Cite

The effect of proximate emotions on risk perceptions is of central importance to criminal decision‐making theory, but has been understudied. We investigate the role of two integral (situational specific) emotional responses, anger and fear, in a decision‐making context regarding the choice to commit assault. We draw on dual‐process models of information processing and appraisal theory to propose a theoretical model in which integral emotions influence decisions and behavior. Using data from an experiment embedded in a survey to a nationwide sample of adults (N = 804), we test the interrelated roles of anger, fear, and traditional rational choice considerations on the intention to commit assault. We find a strong direct association between emotions and intentions to commit assault. Additionally, anger and fear moderate the effect of cognitive deliberations on behavioral intentions and provide a lens through which to evaluate a criminogenic opportunity.

show abstract

mentioning

confidence: 99%

Fight or flight: Integral emotions and violent intentions

Barnum¹,

Solomon

2019

Criminology

View full text Add to dashboard Cite

show abstract

“…Amazon Mechanical Turk is no longer a novel source of research participants within the social sciences. One benefit of researchers' sustained interest in the platform over time is that a number of studies have accumulated suggesting ways for researchers to avoid common pitfalls and plan around common concerns when using MTurk [12]. In this paper, we add to this literature by proposing a method of sampling from MTurk that alleviates one of the most persistent concerns researchers have voiced about the platform: the problem of participant non-naivete.…”

Section: Discussionmentioning

confidence: 99%

“…Less obviously, repeated exposure to the same measures may cause participants to think harder about some tasks, think less about other tasks (due to boredom), or draw connections between the conditions of between-subjects experiments, either based on memory from previous exposure or debriefing after similar studies. Indeed, there are several theoretical reasons for researchers to be concerned about the effect of non-naivete [12], regardless of whether previous exposure helps or hinders participants within any specific study.…”

Section: Concerns About the Mturk Worker Poolmentioning

confidence: 99%

“…The number of workers on MTurk is just one factor that contributes to non-naivete. Another, well-documented source of non-naivete is the "superworker" problem [11][12]. Superworkers are workers who by virtue of being very active on the platform complete more HITs than their share of the MTurk population.…”

Section: Concerns About the Mturk Worker Poolmentioning

confidence: 99%

“…In particular, one question raised repeatedly in recent years is that of whether MTurk is an overused resource. Along with the knowledge that many researchers rely on MTurk, studies have shown that the average lab samples from a small subset of workers [10], and an even smaller group of "superworkers" completes most studies [11][12]. Combined, these issues amount to a general concern that MTurk participants are too familiar with manipulations and measures common in the social sciences.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Tapped Out or Barely Tapped? Recommendations for How to Harness the Vast and Largely Unused Potential of the Mechanical Turk Participant Pool

Robinson¹,

Rosenzweig²,

Moss³

et al. 2019

Preprint

View full text Add to dashboard Cite

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool.Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

show abstract

Examining the effects of memory specificity and perceptual load on susceptibility to misleading information

Farina

Greene

2020

Applied Cognitive Psychology

View full text Add to dashboard Cite

Summary Our study aimed to examine the role of perceptual load in eyewitness memory and susceptibility to misinformation and establish whether trait‐based memory specificity protects against misinformation. Participants (n = 264) viewed a video depicting a crime and completed a memory questionnaire immediately afterwards and 1 week later. Memory specificity was measured using the Autobiographical Memory Test (AMT). Higher AMT scores were associated with better memory accuracy. Performance was worse in the high load compared to the low load condition at immediate recall. However, this effect was not seen for every question and load did not influence eyewitness identifications. To test the possibility that load effects were not fully captured by the questionnaire in experiment 1, we conducted a second experiment (n = 120) where we systematically manipulated misinformation about central and peripheral details. We found no effects. Our findings suggest that high perceptual load enhances eyewitness suggestibility, while specific autobiographical memory protects against misinformation.

show abstract

Common Concerns with MTurk as a Participant Pool: Evidence and Solutions

Abstract: In this chapter, we outline the common concerns with MTurk as a participant pool, review the evidence for those concerns, and discuss solutions. We close with a Table of considerations that researchers should make when fielding a study on MTurk

Cited by 160 publications

References 93 publications

Fight or flight: Integral emotions and violent intentions

Fight or flight: Integral emotions and violent intentions

Tapped Out or Barely Tapped? Recommendations for How to Harness the Vast and Largely Unused Potential of the Mechanical Turk Participant Pool

Examining the effects of memory specificity and perceptual load on susceptibility to misleading information

Contact Info

Product

Resources

About