Methods to detect low quality data and its implication for psychological research

Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices

“…Despite Briones and Benham's (2017) findings that very few differences exist between crowd-sourced and traditionally recruited or convenience samples, researchers have provided evidence that distant participant samples are more likely to complete surveys with less care than are local participant samples, irrespective of possible financial compensations (Litman, Robinson, & Rosenzweig, 2015). In addition, crowd-sourced samples are likely to consist of people who participate regularly in surveys, making them overrepresented (Buchanan & Scofield, 2018;Chandler, Mueller, & Paolacci, 2014). In worse-case scenarios, the presence of fraudulent respondents may be suspected (Chandler & Paolacci, 2017).…”

“…Users may freely download several pieces of software, such as survey bots or automated form-fillers, from the Internet. Such automated form-fillers can be easily programmed (e.g., Buchanan & Scofield, 2018). Liu and Wronski (2018) found that Captcha-type questions represent a promising trap technique to capture inattentive respondents; nonetheless, impostors are likely to bypass such steps by responding to the Captcha and then running automated form-fillers.…”

“…Malicious programs are likely to provide random response sets-namely, uniform deviates (Buchanan & Scofield, 2018;Holden, Wheeler, & Marjanovic, 2012;Meade & Craig, 2012). The main effects of random responding cover the mean level and variance in total scores (Osborne & Blanchard, 2011) and correlations (DeSimone, DeSimone, Harms, & Wood, 2018;Holtzman & Donnellan, 2017;Huang, Liu, & Bowling, 2015), including the factor structure of a psychological test or its reliability (DeSimone et al, 2018), which is at high risk of affecting both a study's representativeness and validity.…”

“…& External indices External indices are additional measures collected with the questionnaire that are not a part of the questionnaire. These indices include response time (Buchanan & Scofield, 2018;Curran, 2016;Meade & Craig, 2012;Niessen et al, 2016), bogus items (Curran, 2016;Meade & Craig, 2012;Niessen et al, 2016), explicit instructed response items (Niessen et al, 2016), and click count, when data are collected electronically (Buchanan & Scofield, 2018).…”

LAB: Linguistic Annotated Bibliography – a searchable portal for normed database information

“…First, Amazon's Mechanical Turk, an online crowdsourcing avenue that allows researchers to pay users to complete questionnaires, can be a reliable, diverse participant pool made available at very low cost (Buhrmester, Kwang, & Gosling, 2011;Mason & Suri, 2012). Researchers can prescreen for specific populations, as well as post-screen surveys for incomplete or inappropriate responses (Buchanan & Scofield, 2018), thus saving time and money with the elimination of poor data. Because of the popularity of Mechanical Turk, large amounts of data can be collected in shorter time periods than traditional experiments.…”