Is it Ethical to Use Mechanical Turk for Behavioral Research? Relevant Data from a Representative Survey of MTurk Participants and Wages

Abstract

To understand human behavior, social scientists need people and data. In the last decade, Amazon’s Mechanical Turk (MTurk) emerged as a flexible, affordable, and reliable source of human participants and was widely adopted by academics. Yet despite MTurk’s utility, some have questioned whether researchers should continue using the platform on ethical grounds. The brunt of their concern is that people on MTurk are financially insecure, subjected to abuse, and earning inhumane wages. We investigated these issues with two random and representative surveys of the U.S. MTurk population (N = 4,094). The surveys revealed: 1) the financial situation of people on MTurk mirrors the general population, 2) the vast majority of people do not find MTurk stressful or requesters abusive, and 3) MTurk offers flexibility and benefits that most people value above more traditional work. In addition, people reported it is possible to earn about 9 dollars per hour and said they would not trade the flexibility of MTurk for less than 25 dollars per hour. Altogether, our data are important for assessing whether MTurk is an ethical place for behavioral research. We close with ways researchers can promote wage equity, ensuring MTurk is a place for affordable, high-quality, and ethical data.

The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers

“…Dennis et al (2019) found that this new surge of low-quality workers is characterized by worker IP addresses that can be traced to a certain type of Internet service provider (ISP) known as virtual private servers. Furthermore, there are serious suspicions that these accounts are operated by people from outside the United States, mainly from India (Moss & Litman, 2018) and Venezuela (Kennedy et al, 2019). Therefore, participation in this study was limited to U.S. workers only and applied a newly designed tool to flag MTurk workers whose IP address was suspicious of malicious activity and/or traced to a non-U.S. location (Prims et al, 2018).…”

Trust Toward Robots and Artificial Intelligence: An Experimental Approach to Human–Technology Interactions Online

“…Data were collected in April 2019 from American participants (N = 1077, 50.60% female, M age = 37.39 years, SD age = 11.42 years) who were recruited via Amazon's Mechanical Turk, which is considered a reliable source to obtain research participants in the United States (Buhrmester et al, 2011;Paolacci and Chandler, 2014;Huff and Tingley, 2015). Recent analysis also showed that the financial situation of MTurk participants mirrors that of the United States and that respondents do not find requesters abusive (Moss et al, 2020). Some concerns have arisen, however, due to non-United States residents trying to access the surveys intended for United States residents only.…”

Refutation Texts: A New Approach to Changing Public Misconceptions About Education Policy

“…Recent concerns were raised about the integrity of the data from MTurk (Litman, 2018), suggesting that that “bots” (i.e., automated fake accounts) were being used to answer surveys with fake data. This issue was shown to be rare to non-existent in a recent analysis (Moss & Litman, 2018). Instead, cultural and linguistic barriers were found to drive poor responses.…”