Quantifying the Distorting Effect of Rapid Guessing on Estimates of Coefficient Αlpha

Abstract

An underlying threat to the validity of reliability measures is the introduction of systematic variance in examinee scores from unintended constructs that differ from those assessed. One construct-irrelevant behavior that has gained increased attention in the literature is rapid guessing (RG), which occurs when examinees answer quickly with intentional disregard for item content. To examine the degree of distortion in coefficient alpha due to RG, this study compared alpha estimates between conditions in which simulees engaged in full solution (i.e., do not engage in RG) versus partial RG behavior. This was done by conducting a simulation study in which the percentage and ability characteristics of rapid responders as well as the percentage and pattern of RG were manipulated. After controlling for test length and difficulty, the average degree of distortion in estimates of coefficient alpha due to RG ranged from -.04 to .02 across 144 conditions. Although slight differences were noted between conditions differing in RG pattern and RG responder ability, the findings from this study suggest that estimates of coefficient alpha are largely robust to the presence of RG due to cognitive fatigue and a low perceived probability of success.

The Impact of Rapid Guessing on Model Fit and Factor-Analytic Reliability: An Exploratory Analysis

“…In this research, we focus on the response time thresholds method due to its ease of implementation, ubiquity in operational settings , and unobtrusiveness. Detailed overviews of the mixture modeling framework can be found in Rios, Abulela, and Deng (2022).…”

“…Note that responses classified as solution behavior (i.e., non-rapid guessing responses) are not subject to the corrective methods discussed in this section (i.e., SB responses are left unaltered). We restrict our attention to four commonly used corrective methods: naïve scoring, penalized scoring, effort-moderated scoring (EM), and EM scoring with imputation (EM-I) (see Rios, Abulela, and Deng (2022) for a comprehensive review).…”

“…An advantage of this approach is that it adjusts for the inflation in scores due to the random guessing strategies examinees engage in. An assumption of penalized scoring is that examinees engage in RG on a given item because the item difficulty is greater relative to their ability (Rios, Abulela, & Deng, 2022). In other words, the penalized scoring approach assumes that, had the examinee not engaged in RG, said examinee would have likely gotten the item wrong anyways because they do not possess the ability needed to successful solve the problem.…”