The MTurkification of Social and Personality Psychology

Anderson, Craig A.; Allen, Johnie J.; Plante, Courtney N.; Quigley‐McBride, Adele; Lovett, Alison; Rokkum, Jeffrey N.

doi:10.1177/0146167218798821

Cited by 105 publications

(94 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, even if one could be convinced that the ethical problems raised by the invisibility of the participants, and by the conflation of research with labor, could be adequately addressed, it may be that Turkers are quickly becoming an un-usable population. More broadly, in the race to publish as many high-impact studies as possible, researchers are incentivized to prioritize data collection, potentially influencing the types of research and theories that get conducted (e.g., Anderson, Allen, Plante, Quigley-McBride, Lovett, & Rokkum, 2018).…”

Section: Resultsmentioning

confidence: 99%

Invisible Participants: Recommendations for Researchers Using Crowdsourcing Platforms for Human Subjects Research

Morrissey¹,

Yamasaki²,

Levitan³

2019

Preprint

View full text Add to dashboard Cite

Increasingly, behavioral scientists rely on crowdsourcing platforms (e.g., Amazon’s MTurk) to collect large datasets rapidly and inexpensively. However, by heightening the distance between participants and researchers, these platforms render the participants “invisible,” and thereby raise ethical concerns related to participant autonomy. Moreover, because many participants use these platforms to earn a living, researchers must consider issues of fair compensation, and the relationship between employment and research participation. We present an overview of ethical considerations raised by the invisibility of crowdsourced research participants, including recommendations for researchers to consider in deciding whether and how to utilize these platforms. Specifically, we recommend that researchers: explore alternatives that distinguish research from employment; strengthen participant-research communication, including by recognizing and utilizing participant expertise in study design; and, explore alternatives that do not require “payment,” or, barring that, determine and pay appropriate compensation. We conclude by locating these ethical considerations in the context of broader calls for more thoughtful data collection.

show abstract

Section: Resultsmentioning

confidence: 99%

Invisible Participants: Recommendations for Researchers Using Crowdsourcing Platforms for Human Subjects Research

Morrissey¹,

Yamasaki²,

Levitan³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…It remains an open question, however, as to whether our results are indicative of the broader American population or if they are particular to Americans on M-Turk. This highlights the drawbacks of homogenous sampling in psychological studies drawing on mainly Americans (Henrich, Heine, & Norenzayan, 2010) with an increasing reliance on M-Turk (Anderson et al, 2019). Future research would benefit from the specific sampling of similarly religious but highly liberal and highly conservative populations (on and off M-Turk).…”

Section: Discussionmentioning

confidence: 99%

How is analytical thinking related to religious belief? A test of three theoretical models

Baimel¹,

White²,

Norenzayan³

2019

Preprint

View full text Add to dashboard Cite

A growing literature has documented a negative association between analytical thinking style and belief in God. However, the replicability, magnitude, and theoretical importance of this correlation has recently been debated. Moreover, the existing literature has not examined distinct psychological accounts of this relationship. In Study 1, we (1) tested the replicability of the correlation and assessed its magnitude in a large sample (N = 5284; comprising of undergraduate students at a Canadian university, and broader samples of Canadians, Americans and Indians); and (2) tested three distinct theoretical accounts of how cognitive style might come to be related to a diverse set of religious beliefs including belief in God, in karma, and in witchcraft. The first, the dual process model, posits that analytical thinking is inversely related to belief in God and in other supernatural entities. The second, the expressive rationality model, posits that analytical thinking is specifically recruited in supporting already-held beliefs in an identity-protective manner. And the third, the counter-normativity rationality model, posits that analytical thinking is recruited to question beliefs supported by prevailing cultural norms. We tested specific predictions derived from these models regarding the association between analytic thinking and religious beliefs in a Bayesian framework. In Study 2, we tested the replicability of our results in a re-analysis of previously-published data. We conclude that whereas the counter-normativity rationality model was contradicted by the data, both the dual process and expressive rationality models received limited empirical support.

show abstract

“…Researchers typically exploit one of the online labour markets, such as MTurk or Prolific. Research in the Social Sciences, in particular, has seen an increase in the use of MTurk and similar online platforms [6]. These online labour markets all suffer from various issues and challenges [14,34].…”

Section: Creativity Studies On Crowdsourcing Platformsmentioning

confidence: 99%

Design recommendations for augmenting creative tasks with computational priming

Oppenlaender

Hosio

2019

Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia

View full text Add to dashboard Cite

Supporting creativity is a grand challenge in HCI. A critical component of creativity is the ability for divergent thinking, and divergent thinking can be fostered through looking at the problem through the lens of a different person, by assuming a role. Prior work found that assuming a role and affective stimulation with images may lead individuals to be more creative. In this work, we investigate the use of roles in stimulating the creativity of individuals in two complementary studies. In the first study, we implemented an online instrument for augmenting creativity with roles and images, and recruited crowd workers (n = 60) to complete a divergent thinking task while assuming a role. Interestingly, and in contrast to earlier findings, our analysis could not confirm the computational priming having an effect on the outcome of a small batch of creative tasks. In the second study, we observed the effect of roles on the ideation process of individuals when they reach an impasse in the flow of ideas. Our complementary studies highlight that adopting roles can help when one runs out of ideas, but this is not a silver bullet for improving divergent thinking, especially in online crowdsourcing environments that are increasingly being used for experiments and data collection in behavioural science. Our work informs the design of future crowd-powered creativity support tools and contributes a timely case study to the body of literature in the growing field of creativity support online.

show abstract

The MTurkification of Social and Personality Psychology

Cited by 105 publications

References 19 publications

Invisible Participants: Recommendations for Researchers Using Crowdsourcing Platforms for Human Subjects Research

Invisible Participants: Recommendations for Researchers Using Crowdsourcing Platforms for Human Subjects Research

How is analytical thinking related to religious belief? A test of three theoretical models

Design recommendations for augmenting creative tasks with computational priming

Contact Info

Product

Resources

About