A Comparison of Robust Likelihood Estimators to Mitigate Bias From Rapid Guessing

Abstract

Rapid guessing (RG) behavior can undermine measurement property and score-based inferences. To mitigate this potential bias, practitioners have relied on response time information to identify and filter RG responses. However, response times may be unavailable in many testing contexts, such as paper-and-pencil administrations. When this is the case, self-report measures of effort and person-fit statistics have been used. These methods are limited in that inferences concerning motivation and aberrant responding are made at the examinee level. As test takers can engage in a mixture of solution and RG behavior throughout a test administration, there is a need to limit the influence of potential aberrant responses at the item level. This can be done by employing robust estimation procedures. Since these estimators have received limited attention in the RG literature, the objective of this simulation study was to evaluate ability parameter estimation accuracy in the presence of RG by comparing maximum likelihood estimation (MLE) to two robust variants, the bisquare and Huber estimators. Two RG conditions were manipulated, RG percentage (10%, 20%, and 40%) and pattern (difficulty-based and changing state). Contrasted to the MLE procedure, results demonstrated that both the bisquare and Huber estimators reduced bias in ability parameter estimates by as much as 94%. Given that the Huber estimator showed smaller standard deviations of error and performed equally as well as the bisquare approach under most conditions, it is recommended as a promising approach to mitigating bias from RG when response time information is unavailable.

Comparing Person-Fit and Traditional Indices Across Careless Response Patterns in Surveys

“…This is also true for response time. While we did not simulate rapid responding as a carelessness pattern, future studies may wish to focus on simulating observational indices such as response time in a similar manner that has been undertaken by those evaluating rapid responses in other assessment settings (see Rios, 2022).…”

The Impact of Rapid Guessing on Model Fit and Factor-Analytic Reliability: An Exploratory Analysis

“…Most RG identification procedures rely on the ability to obtain response time information at the item level. Recently, however, RG identification methods that do not rely on response time information have been developed (Rios, 2022) and these have been found to be especially useful in contexts in which response times may not be available (e.g., paper-andpencil administrations). In this research, we focus on the response time thresholds method due to its ease of implementation, ubiquity in operational settings , and unobtrusiveness.…”

“…NER is a behavior characterized by random responding without consideration for the item content (Rios, 2022;Wise & DeMars, 2005). According to the expectancy-value theory of achievement motivation (e.g., Pintrich & Schunk, 2002), the likelihood an examinee engages in NER is dictated by two principal factors: (1) the examinee's perception of the importance and usefulness of the presented item (i.e., task value; TV), and (2) the recognition that the examinee does not possess the requisite background needed to successfully solve a given problem (i.e., perceived probability of success; PPS).…”