Variance Estimation for Naep Data Using a Resampling‐based Approach: An Application of Cognitive Diagnostic Models

Abstract

This paper presents an application of a jackknifing approach to variance estimation of ability inferences for groups of students, using a multidimensional discrete model for item response data. The data utilized to demonstrate the approach come from the National Assessment of Educational Progress (NAEP). In contrast to the operational approach used in NAEP, where plausible values are used to make ability inferences, the approach presented in this paper reestimates all parameters of the model, and makes ability inferences based on replicate samples of the jackknife without using plausible values.Results of the standard errors are presented for estimates of group means, total means, and other statistics used in official reporting by NAEP. Differences in results between this approach and the operational approach are discussed.

Measuring Growth in a Longitudinal Large-Scale Assessment with a General Latent Variable Model

“…Standard errors of item parameters were estimated using a jackknife resampling approach. Details on the procedure used for variance estimation can be found in Hsieh, Xu, and von Davier (2009). When applying the GDM to longitudinal data, the probability of a response depends on item-difficulty and occasion-specific parameters.…”

Linking Errors in Trend Estimation in Large‐scale Surveys: A Case Study

“…Note that this research has used the comprehensive reestimation of all parameters of the multiple group IRT model as described in Hsieh, Xu, and von Davier (2009). A less comprehensive approach like the one currently used operationally may have resulted in a larger difference between full item set and double jackknife.…”

Large-Scale Group-Score Assessment

“…Because of the effects of a complex sample design, the true sampling variability is usually larger than a simple random sampling. More detailed information is available in reports by Rust (1992, 1993), Johnson and King (1987), and Hsieh et al (2009).…”

“…Each of these replicates consists of all PSU data except for one, in which one school is randomly removed from the estimate and the other is weighted doubly. The methodology for NAEP was described, for example, by E. G. Johnson and Rust (1992), and von Davier et al (2006), and a possible extension was discussed by Hsieh et al (2009).…”

“…For example, Hsieh et al (2009) explored a resampling-based approach to variance estimation that makes ability inferences based on replicate samples of the jackknife without using plausible values.…”