Consistent Estimation of Rasch Item Parameters and Their Standard Errors Under Complex Sample Designs

Abstract

U.S. state educational testing programs administer tests to track student progress and hold schools accountable for educational outcomes. Methods from item response theory, especially Rasch models, are usually used to equate different forms of a test. The most popular method for estimating Rasch models yields inconsistent estimates and relies on ad hoc adjustments to obtain good approximations. Furthermore, psychometricians have paid little attention to the estimation of effective standard errors for Rasch models, especially under complex sample designs. This article presents a computationally efficient, statistically consistent estimator for Rasch models, based on a nonparametric marginal maximum likelihood approach, along with complete, designconsistent estimators of the standard error, based on the full information matrix and including covariance terms among items, covariances between items, and parameters of the distribution of the latent trait. Simulations support the consistency of the estimators in both simple random samples and more realistic multistage samples. Index terms: Rasch model, standard error, nonparametric marginal maximum likelihood, equating, item response theory State educational testing programs periodically administer tests to students to track individual student progress and to hold teachers, schools, or districts accountable for educationally meaningful outcomes. Operationally, running these programs requires the periodic release of items, both to avoid their accidental or malicious disclosure and to inform the public of the types of tasks students are expected to perform. Replacement items, and even whole replacement test forms, must be linked to the same underlying scale as the tests or items that they replace.In the United States, state testing programs rely on a very limited set of statistical measurement models to link test items to a common scale. Perhaps the most common measurement models belong to the Rasch family of statistical models. Several characteristics of Rasch models (discussed briefly below) render them particularly attractive for programs that will receive public scrutiny.Within the United States, current practice in the estimation of Rasch models lags behind the statistical research on the subject in two ways. First, the most popular method and software for estimating Rasch models in U.S. state testing programs yield inconsistent estimates and rely on ad hoc adjustments to obtain good approximations (see Wright & Douglas, 1977). Second, psychometricians have paid remarkably little attention to the estimation of effective standard errors (SEs) for Rasch models, especially under the complex sample designs typically found in state testing programs.This article presents a computationally efficient, statistically consistent estimator for Rasch models, along with complete, design-consistent estimators of the SE. The proposed estimator is based on a nonparametric marginal maximum likelihood approach (NPMML) to estimation. The SE developed here is complete in that it is based...

A Multilevel Testlet Model for Dual Local Dependence

“…Other researchers have included person‐level covariates to improve the estimation of item and person parameters (e.g., Mislevy, 1987; Mislevy & Bock, 1989). Another attempt was made by Cohen, Chan, Jiang, and Seburn (2008) to use a nonparametric method to study Rasch modeling under the complex sample designs typically found in state testing programs. In addition, multilevel IRT models (Johnson & Jenkins, 2005; Li, Oranje, & Jiang, 2009) have been applied in large‐scale survey assessments such as the National Assessment of Educational Progress.…”

Impact of Design Effects in Large-Scale District and State Assessments

“…The normalized weights,ŵ hij,R , are found by dividing 3 iAM is free psychometric software available from the American Institutes for Research (AIR) and Jon Cohen, and can be downloaded at http://am.air.org. The software uses an item response theory module (iAM) for parameter estimation of all major item response theory models, along with design consistent estimates of standard errors (Cohen, Chan, Jiang, & Seburn, 2008). The standard errors are estimated using Taylor series approximations, which take into account student weights, stratification information, and clustering (Binder, 1983;Woodruff, 1971).…”

“…However, the procedures and formula provided by Kolen and Brennan assume simple random sampling so they do not capture the additional error in the standard error of equating caused by design effects in large-scale assessment sampling. One example of research that explicitly incorporates the design effect in the standard error of equating was reported by Cohen et al (2008).…”

Estimating the Multilevel Rasch Model: With thelme4Package

“…These important dependencies are commonly ignored, resulting in estimates that may be inconsistent and standard errors that do not adequately characterize the true variance. One option to consider in the presence of a non-zero design effect (Kish 1965) is to regard the point estimates as retaining some utility, but construct robust standard errors in recognition of the fact that correlated observations provide less information than an equivalent number from a simple random sample (Binder 1983;Cohen, Jiang, and Seburn 2005).…”