Weighting Test Samples in Irt Linking and Equating: Toward an Improved Sampling Design for Complex Equating

Abstract

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. SAT is a registered trademark of the College Board.i Abstract Several factors could cause variability in item response theory (IRT) linking and equating procedures, such as the variability across examinee samples and/or test items, seasonality, regional differences, native language diversity, gender, and other demographic variables. Hence, the following question arises: Is it possible to select optimal samples of examinees so that the IRT linking and equating can be more precise at an administration level as well as over a large number of administrations? This is a question of optimal sampling design in linking and equating. To obtain an improved sampling design for invariant linking and equating across testing administrations, we applied weighting techniques to yield a weighted sample distribution that is consistent with the target population distribution. The goal is to obtain a stable StockingLord test characteristic curve (TCC) linking and a true-score equating that is invariant across administrations. To study the weighting effects on linking, we first selected multiple subsamples from a data set. We then compared the linking parameters from subsamples with those from the data and examined whether the linking parameters from the weighted sample yielded smaller mean square errors (MSE) than those from the unweighted subsample. To study the weighting effects on true-score equating, we also compared the distributions of the equated scores.Generally, the findings were that the weighting produced good results.

Effect of Statistically Matching Equating Samples for Common‐Item Equating

“…With test takers' multiple background variables that are statistically related to the test scores, such as age, gender, education, and testing experiences, the weighted equating proposed in this study can still be implemented, but with a more sophisticated weighting process. Some possibilities are propensity score matching, DAG, as in Livingston (2014), the minimum discriminant information method as in Haberman (1984), or poststratification ranking as in Qian et al (2013).…”

“…With simulation studies based on real data, they showed that the equating based on common items could be improved by incorporating the PEG adjustment procedure into the NEAT process. With common item designs, Qian et al (2013) studied the effects of weighting test takers using item response theory (IRT) true‐score equating. They found that the scale transformation parameters for converting the item parameter estimates from one scale to another were more stable with weighted equating than with unweighted equating.…”

Variance Estimation With Complex Data and Finite Population Correction—A Paradigm for Comparing Jackknife and Formula‐Based Methods for Variance Estimation

“…For GJRR, a replicate sample is formed by randomly dropping one group of cases from the sample and making adjustment to weights based on statistical techniques such as minimum discriminant information adjustment (Haberman, 2015), raking, and poststratification (Qian et al, 2013). Let J be the number of total replicate samples.…”

Contributions to Score Linking Theory and Practice

“…Results indicated that although this procedure adjusts for group differences, it does not reduce the ability difference between the new and old form samples enough to warrant use. Qian et al (2013) used techniques for weighting observations to yield a weighted sample distribution that is consistent with the target population distribution to achieve true-score equatings that are more invariant across administrations than those obtained with unweighted samples.…”