Detecting careless respondents in web-based questionnaires: Which method to use?

Niessen, A. Susan M.; Meijer, Rob R.; Tendeiro, Jorge N.

doi:10.1016/j.jrp.2016.04.010

Cited by 155 publications

(221 citation statements)

References 28 publications

(67 reference statements)

Supporting

Mentioning

196

Contrasting

Unclassified

Order By: Relevance

“…The use of response times to detect the aberrant answering of assessment items is well known 8–11,13–20,51–57. To our knowledge, no previous research has evaluated the utility of SOAPP-R completion times to predict ADB, despite the fact that respondent deception has been identified as a concern for this questionnaire 7.…”

Section: Discussionmentioning

confidence: 99%

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R)

Finkelman

Kulich

Butler

et al. 2016

JPR

View full text Add to dashboard Cite

BackgroundRespondents’ scores to the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R) have been shown to be predictive of aberrant drug-related behavior (ADB). However, research is lacking on whether an individual’s completion time (the amount of time that he/she takes to finish the screener) has utility in predicting ADB, despite the fact that response speed has been useful in predicting behavior in other fields. The purpose of this study was to evaluate the degree to which SOAPP-R completion time is predictive of ADB.Materials and methodsThis retrospective study analyzed completion-time data from 82 adult emergency department patients who completed the SOAPP-R on a tablet computer. The utility of SOAPP-R completion times in predicting ADB was assessed via logistic regression and the area under the curve (AUC) statistic. An external measure of ADB using Prescription Drug Monitoring Program data defined ADB to have occurred in individuals with at least four opioid prescriptions and at least four prescribers in 12 months.ResultsAlthough there was a slight trend for individuals with greater completion times to have greater odds of ADB (odds ratio 1.004 in simple logistic regression), the association between SOAPP-R completion time and ADB was not statistically significant in either simple logistic regression (P=0.307) or multiple logistic regression adjusting for SOAPP-R score (P=0.419). AUC values for the prediction of ADB using completion time alone, SOAPP-R score alone, and both completion time and SOAPP-R score were 0.63, 0.64, and 0.65, respectively.ConclusionThere was no significant evidence that SOAPP-R completion times were predictive of ADB among emergency department patients. However, the AUC value for completion times was only slightly less than that for SOAPP-R total scores.

show abstract

Section: Discussionmentioning

confidence: 99%

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R)

Finkelman

Kulich

Butler

et al. 2016

JPR

View full text Add to dashboard Cite

show abstract

“…In the literature, some suggest that responses should be removed once identified as aberrant (e.g. Curran, 2016;Niessen, Meijer, & Tendeiro, 2016). However, data removal can be controversial.…”

Section: Methodsmentioning

confidence: 99%

Robust Maximum Marginal Likelihood (RMML) Estimation for Item Response Theory Models

Hong¹,

Cheng²

2018

Preprint

View full text Add to dashboard Cite

Self-report data are common in psychological and survey research. Unfortunately, many of these samples are plagued with careless responses due to unmotivated participants. The purpose of this study is to propose and evaluate a robust estimation method in order to detect careless, or unmotivated, responders while leveraging Item Response Theory (IRT) person fit statistics. First, we outline a general framework for robust estimation specific for IRT models.Subsequently, we conduct a simulation study covering multiple conditions to evaluate the performance of the proposed method. Ultimately, we show how robust maximum marginal likelihood (RMML) estimation significantly improves detection rates for careless responders and reduce bias in item parameters across conditions. Furthermore, we apply our method to a real dataset to illustrate the utility of the proposed method. Our findings suggest that robust estimation coupled with person fit statistics offers a powerful procedure to identify careless respondents for further review, and to provide more accurate item parameter estimates in presence of careless responses.

show abstract

“…Some recently proposed screening techniques could not be included in the present analysis due to the nature of the sample and data. Guttman errors and IRT-informed response probabilities are gaining popularity (see, Curran, 2016;Niessen, Meijer, & Tendeiro, 2016). However, as these techniques are computed using various operationalizations of item difficulty and/or discrimination, they are most appropriate for use with unidimensional scales containing many items (Meijer, Monelaar, & Sijtsma, 1994).…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

“…Absent a much larger sample or normative data, estimates of item parameters would be too unstable to be useful for computing these indices (De Ayala & Sava-Bolesta, 1999;DeMars, 2003). Niessen et al (2016) examined the cutoff values and relationships of these indices with some of the techniques assessed in the current paper. Future research should examine the distributional characteristics of these techniques as well as their potential impact on item inter-relationships or the results of statistical analyses (research questions 3, 4, and 5 above).…”

Section: Limitations and Future Directionsmentioning

confidence: 99%

Dirty Data: The Effects of Screening Respondents Who Provide Low-Quality Data in Survey Research

2017

View full text Add to dashboard Cite

The purpose of this study is to empirically address questions pertaining to the effects of data screening practices in survey research. This study addresses questions about the impact of screening techniques on data and statistical analyses. It also serves an initial attempt to estimate descriptive statistics and graphically display the distributions of popular screening techniques. Data were obtained from an online sample who completed demographic items and measures of character strengths (N = 307). Screening indices demonstrate minimal overlap and differ in the number of participants flagged. Existing cutoff scores for most screening techniques seem appropriate, but cutoff values for consistency-based indices may be too liberal. Screens differ in the extent to which they impact survey results. The use of screening techniques can impact inter-item correlations, inter-scale correlations, reliability estimates, and statistical results. While data screening can improve the quality and trustworthiness of data, screening techniques are not interchangeable. Researchers and practitioners should be aware of the differences between data screening techniques and apply appropriate screens for their survey characteristics and study design. Low-impact direct and unobtrusive screens such as self-report indicators, bogus items, instructed items, longstring, individual response variability, and response time are relatively simple to administer and analyze. The fact that data screening can influence the statistical results of a study demonstrates that low-quality data can distort hypothesis testing in organizational research and practice. We recommend analyzing results both before and after screens have been applied.

show abstract

Detecting careless respondents in web-based questionnaires: Which method to use?

Cited by 155 publications

References 28 publications

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R)

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R)

Robust Maximum Marginal Likelihood (RMML) Estimation for Item Response Theory Models

Dirty Data: The Effects of Screening Respondents Who Provide Low-Quality Data in Survey Research

Contact Info

Product

Resources

About

Detecting careless respondents in web-based questionnaires: Which method to use?

Cited by 155 publications

References 28 publications

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain &ndash; revised (SOAPP-R)

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain &ndash; revised (SOAPP-R)

Robust Maximum Marginal Likelihood (RMML) Estimation for Item Response Theory Models

Dirty Data: The Effects of Screening Respondents Who Provide Low-Quality Data in Survey Research

Contact Info

Product

Resources

About

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R)

An investigation of completion times on the Screener and Opioid Assessment for Patients with Pain – revised (SOAPP-R)