Caution! MTurk Workers Ahead—Fines Doubled

Abstract

Landers and Behrend (2015) are the most recent in a long line of researchers who have suggested that online samples generated from sources such as Amazon's Mechanical Turk (MTurk) are as good as or potentially even better than the typical samples found in psychology studies. It is important that the authors caution that researchers and reviewers need to carefully reflect on the goals of research when evaluating the appropriateness of samples. However, although they argue that certain types of samples should not be dismissed out of hand, they note that there is only scant evidence demonstrating that online sources can provide usable data for organizational research and that there is a need for further research evaluating the validity of these new sources of data. Because the target article does not directly address the potential problems with such samples, we will review what is known about collecting online data (with a particular focus on MTurk) and illustrate some potential problems using data derived from such sources.

Is Mechanical Turk the Answer to Our Sampling Woes?

“…Unlike more traditional samples mentioned in the focal article, MTurk samples do tend to be more diverse in terms of occupations. A more recent study by Harms and DeSimone (2015) reported only a .12 correlation for industry representativeness when comparing MTurk samples and Department of Labor statistics. In another study, Downs and colleagues (2010) reported a similar pattern with science, engineering, and information technology (24.71%); business, management, and financial services (14.36%); administrative support (10.27%); education (9%); art, writing, and journalism (6.34%); service (5.75%); medical (3%); skilled labor (1.75%); and legal services (1.25%).…”

Identity economics meets identity leadership: Exploring the consequences of elevated CEO pay

“…Participants and design. Six-hundred-and-twenty-seven US residents participated in the present study after being recruited via Amazon's Mechanical Turk (for discussions of strengths and limitations of this platform, see Harms & DeSimone, 2015;Landers & Behrend, 2015). Thirty-seven participants who failed to respond to three control questions as requested ("This is a control question-please select '1'/ '2' / 'strongly disagree'") were excluded, leaving a total of 590 participants in the final sample (Mage=35.35 years; SD=10.56; 46% female).…”

Attention! Cosmopolitans ahead – They may not be as creative as they think!

“…Although our findings question how well certain assumptions about creativity, such as self‐expectations for creativity and self‐reported creative work involvement, can be applied to understanding creative processes, the study has limitations that require modest interpretations of the results. First, we recognize that some researchers view MTurk data with scepticism (Harms & DeSimone, ; Landers & Behrend, ). Anticipating this, we used a number of screening tools to collect the highest quality data as possible.…”

“…This workforce is distributed throughout the globe; researchers can limit this distribution to a particular region. Participants were screened based on a number of items including location (US only), working status and two items that provide evidence of the individual's motivations: having completed less than 5,000 total approved studies and a 98% approval rating; in order to ensure a high‐quality sample (Harms & DeSimone, ). After conducting outlier analysis and dropping incomplete, and thus, unusable data, the resulting sample was N = 151.…”