Long‐Term Impact of Valid Case Criterion on Capturing Population‐Level Growth Under Item Response Theory Equating

Abstract

Using simulated data, this study examined the impact of different levels of stringency of the valid case inclusion criterion on item response theory (IRT)‐based true score equating over 5 years in the context of K–12 assessment when growth in student achievement is expected. Findings indicate that the use of the most stringent inclusion criterion generally yielded the most accurate results when overall root mean square error (RMSE) and bias were considered under both zero‐growth and growth conditions, for both one‐parameter logistic (1PL) and three‐parameter logistic (3PL) IRT models, and for both fixed common item parameter (FCIP) and test characteristic curve (TCC) scaling methods. The positive impact of applying the most stringent valid case inclusion criterion was more salient with the 3PL model, under which greater classification accuracy was observed.

Effect of Item Parameter Drift in Mixed Format Common Items on Test Equating

“…The RMSE value was calculated using the equation below where θ stands for true ability, θ* stands for estimated true ability, and f stands for ability frequency (Deng & Monfils, 2017;Keller & Keller, 2011). Since this was a simulation study based on replications, the means of RMSE values were calculated.…”

Automated Essay Scoring Effect on Test Equating Errors in Mixed-format Test

“…The following equations can be used when calculating SEE (equality 4), BIAS (equality 5) and RMSE (equality 6) values based on IRT. The resources of Deng and Monfils (2017) and Keller and Keller (2011) were used for equations. Ɵ is the ability of the individual i, Ɵ is the ability of the individual i estimated by the equating method used, and N is the sample size:…”