Parameter Estimation Accuracy of the Effort-Moderated IRT Model Under Multiple Assumption Violations

Abstract

As low-stakes testing contexts increase, low test-taking effort may serve as a serious validity threat. One common solution to this problem is to identify noneffortful responses and treat them as missing during parameter estimation via the Effort-Moderated IRT (EM-IRT) model. Although this model has been shown to outperform traditional IRT models (e.g., 2PL) in parameter estimation under simulated conditions, prior research has failed to examine its performance under violations to the model’s assumptions. Therefore, the objective of this simulation study was to examine item and mean ability parameter recovery when violating the assumptions that noneffortful responding occurs randomly (assumption #1) and is unrelated to the underlying ability of examinees (assumption #2). Results demonstrated that, across conditions, the EM-IRT model provided robust item parameter estimates to violations of assumption #1. However, bias values greater than 0.20 SDs were observed for the EM-IRT model when violating assumption #2; nonetheless, these values were still lower than the 2PL model. In terms of mean ability estimates, model results indicated equal performance between the EM-IRT and 2PL models across conditions. Across both models, mean ability estimates were found to be biased by more than 0.25 SDs when violating assumption #2. However, our accompanying empirical study suggested that this biasing occurred under extreme conditions that may not be present in some operational settings. Overall, these results suggest that the EM-IRT model provides superior item and equal mean ability parameter estimates in the presence of model violations under realistic conditions when compared to the 2PL model.

Assessing the Accuracy of Parameter Estimates in the Presence of Rapid Guessing Misclassifications

“…RG has been shown to bias measurement properties, such as reliability estimates (e.g., Wise & DeMars, 2009), measurement invariance (e.g., DeMars & Wise, 2010), linking coefficients (e.g., Mittelhaëuser et al, 2015), and item and person parameter estimates (e.g., Rios & Soland, 2020;van Barneveld, 2007). Furthermore, as RG is generally associated with underestimation of examinee ability (Silm et al, 2020), it has been documented to bias treatment effects (e.g., Osborne & Blanchard, 2011;Liu et al, 2015), achievement gains (e.g., Wise & DeMars, 2010), value-added estimates of teacher effectiveness (Wise et al, 2013), and subgroup comparisons (e.g., Debeer et al, 2014).…”

“…Concerning the former assumption, prior research has suggested that it may be untenable in practice given that RG has been found to be associated with item position, length, difficulty, and depth of knowledge required (e.g., Wise, 2020). However, recent evidence indicates that aggregate-level ability estimates may be largely robust to such a violation (Rios & Soland, 2020). In terms of the latter assumption, aggregate-level ability estimates will be biased either positively or negatively if noneffortful responders underlying ability is consistently higher or lower than the average ability of effortful responders.…”

“…In terms of the latter assumption, aggregate-level ability estimates will be biased either positively or negatively if noneffortful responders underlying ability is consistently higher or lower than the average ability of effortful responders. Evidence from operational tests have illustrated that noneffortful responding can occur among examinees with predominately low prior ability (e.g., Kuhfeld & Soland, 2020;Rios et al, 2017), and under such circumstances, biased ability parameter estimates can be obtained via the EM-IRT model (Rios & Soland, 2020).…”

“…Although meeting the assumptions underlying the EM-IRT model is a prerequisite to obtaining accurate item and ability parameter estimates, numerous simulation and applied studies have shown that the use of this model is associated with improved parameter estimation and convergent validity with external variables compared to naïve models that include RG responses in scoring (e.g., Liu et al, 2019;Rios et al, 2017;Rios & Soland, 2020;Wise & DeMars, 2006;Wise & Kingsbury, 2016). Furthermore, apart from establishing a response time threshold to classify RG, this model does not require additional parameter estimation.…”

“…Furthermore, apart from establishing a response time threshold to classify RG, this model does not require additional parameter estimation. Due to this simplicity, it has become one of the most popular models used to study RG (e.g., Rios et al, 2017;Rios & Soland, 2020;Wise & Kingsbury, 2016), and is currently one of the only models to be employed in operational testing contexts to report examinees' RG behavior to score users (see Wise & Kuhfeld, 2020).…”