The Effect of Changing Content on IRT Scaling Methods

Abstract

Equating test forms is an essential activity in standardized testing, with increased importance with the accountability systems in existence through the mandate of Adequate Yearly Progress. It is through equating that scores from different test forms become comparable, which allows for the tracking of changes in the performance of students from one year to the next. This study compares three different item response theory scaling methods (fixed common item parameter, Stocking & Lord, and Concurrent Calibration) with respect to examinee classification into performance categories, and estimation of the ability parameter, when the content of the test form changes slightly from year to year, and the examinee ability distribution changes. The results indicate that calibration methods, especially concurrent calibration, produced more stable results than the transformation method.Equating test forms is an essential activity in standardized testing, with increased importance with the accountability systems in existence through the mandate of adequate yearly progress (AYP). It is through equating that scores from different test forms become comparable, which allows for the tracking of changes in the performance of students from one year to the next. Given that there are different methods for equating, the choice of equating methods could lead to different results, and different inferences about the nature of the achievement of the students in question. As such, choosing the most appropriate equating method for the particular situation is essential. While there may not be an equating method that is best in all situations, studies that examine the context of interest can help inform the choice of methods.In item response theory (IRT), there are primarily two methods of equating: observed score equating and true score equating. Although there are only two popular methods of equating the tests, in the context of IRT, the heart of equating is actually a scaling step in which the parameter estimates from different calibrations are put onto a common scale, or metric. There are several popular scaling techniques that are usually implemented in IRT, and the research regarding these methods is not conclusive. The best method typically depends on the context of the equating. In particular, the degree of equivalence between groups that are used in the scaling and equating appears to be a factor that differentiates scaling methods. In the case of equivalent groups, very little difference in results have been observed while with non-equivalent groups, differences have

The Benefits of Fixed Item Parameter Calibration for Parameter Accuracy in Small Sample Situations in Large‐Scale Assessments

“…Bias depends on the sample size (Hanson & Béguin, 2002; Kang & Petersen, 2012), the number of items with parameters available from previous calibrations (e.g., Arai & Mayekawa, 2011; Kim, Cole, & Mwavita, 2018), the amount of cross‐national DIF (Sachse, Roppelt, & Haag, 2016), and shifts in the latent ability distributions across assessments (e.g., Baldwin, Baldwin, & Nering, 2007; Keller, Keller, & Baldwin, 2007). Keller and Keller (2011, 2015), however, showed that FIPC works best for complex changes in the latent ability distributions and in cases where the content of the assessment changes. Zhao and Hambleton (2017) showed that FIPC was robust against ability shifts across two adjacent assessments.…”

What Affects the Quality of Score Transformations? Potential Issues in True-Score Equating Using the Partial Credit Model

“…If equating studies vary the δ or sometimes the θ distribution, then mostly to challenge the equity requirement, in the form of a shift of the mean, i.e., simulating change or growth, as in Kopp and Jones (2020), Han et al (2012), He et al (2013), or Waterbury and DeMars (2021). The skewness of the distribution of the θ parameter is also sometimes varied, but by preserving good targeting, that is, with overlapping dispersion of the δ and θ , as in Manna and Gu (2019) or Keller and Keller (2015). In any case, Suanthong et al (2000) mention, citing L.…”