Investigating the Impact of Noneffortful Responses on Individual-Level Scores: Can the Effort-Moderated IRT Model Serve as a Solution?

Abstract

Suboptimal effort is a major threat to valid score-based inferences. While the effects of such behavior have been frequently examined in the context of mean group comparisons, minimal research has considered its effects on individual score use (e.g., identifying students for remediation). Focusing on the latter context, this study addressed two related questions via simulation and applied analyses. First, we investigated how much including noneffortful responses in scoring using a three-parameter logistic (3PL) model affects person parameter recovery and classification accuracy for noneffortful responders. Second, we explored whether improvements in these individual-level inferences were observed when employing the Effort Moderated IRT (EM-IRT) model under conditions in which its assumptions were met and violated. Results demonstrated that including 10% noneffortful responses in scoring led to average bias in ability estimates and misclassification rates by as much as 0.15 SDs and 7% respectively. These results were mitigated when employing the EM-IRT model, particularly when model assumptions were met. However, once model assumptions were violated, the EM-IRT model’s performance deteriorated, though still outperforming the 3PL model. Thus, findings from this study show that: (a) including noneffortful responses when using individual scores can lead to potential unfounded inferences and potential score misuse; and (b) the negative impact that noneffortful responding has on person ability estimates and classification accuracy can be mitigated by employing the EM-IRT model, particularly when its assumptions are met.

When Should Individual Ability Estimates Be Reported if Rapid Guessing Is Present?

“…If one is interested in providing descriptive information, the results suggest that the degree of bias is dependent on the interaction between test difficulty and examinee ability, supporting prior findings (e.g., Rios & Soland, 2021b). Therefore, a single criterion cannot be effectively applied to all assessments, score uses, RG patterns, and examinee ability levels.…”

“…For example, under certain conditions, when RG responses comprise only 6% of a data matrix, aggregated examinee ability can be negatively biased by 0.20 standard deviations (Rios et al, 2017). This degree of bias can potentially undermine a number of measurement property and score-based inferences, such as item parameter estimates (van Barnevald, 2007), measurement invariance (Rios, 2021a), proficiency classifications (Rios & Soland, 2021b), treatment effects (e.g., Osborne & Blanchard, 2011), achievement gains (e.g., Wise & DeMars, 2010), and growth estimates (Yildirim-Erbasli & Bulut, 2020).…”

“…Second, employing a general exclusionary rule for all examinees may be ineffective because it ignores the fact that the degree of ability estimate bias due to RG is driven largely by the interaction between item difficulty and theta parameters (Schuster & Yuan, 2011); thus, it is likely that the direction and degree of bias may differ by examinee ability (Rios & Soland, 2021b). As a result of these limitations, it is argued that to better guide decision-making for practitioners, a more systematic approach is needed that can take into consideration multiple factors, including score use as well as assessment and RG characteristics.…”

Investigating the Effect of Differential Rapid Guessing on Population Invariance in Equating

“…3 First, all 50 items in a given test form were divided into quantiles (i.e., 4 bins, items 1–12, 13–25, 26–38, 39–50). Second, for each RG simulee, the number of RG item responses were assigned in increasing magnitude for each bin of items (similar to Rios & Soland, 2021). To reflect reality, rapid guessers were manipulated to engage in RG on a different percentage and set of items, while constraining the mean RG percentage within subgroup (details provided below).…”

To What Degree Does Rapid Guessing Distort Aggregated Test Scores? A Meta-analytic Investigation

“…Based on this latter assumption, proponents of this approach argue that although an examinee may engage in RG, their data may possess valid responses that can be used in estimating ability (e.g., Rios & Soland, 2021b;Wise & Kingsbury, 2016). To this end, a number of Item Response Theory (IRT) models have been proposed (for a review, see Wise & Kingsbury, 2016), with some (e.g., Effort-Moderated IRT model) showing improved estimation accuracy over traditional models that ignore the presence of RG (e.g., Liu et al, 2019;Rios & Soland, 2021a). However, it should be noted that these scoring approaches cannot fully mitigate bias from noneffortful responding, particularly when RG is related to the underlying ability of examinees (e.g., Rios & Soland, 2021a).…”