Linking Errors in Trend Estimation in Large‐scale Surveys: A Case Study

Abstract

One of the major objectives of large‐scale educational surveys is reporting trends in academic achievement. For this purpose, a substantial number of items are carried from one assessment cycle to the next. The linking process that places academic abilities measured in different assessments on a common scale is usually based on a concurrent calibration of adjacent assessments using item response theory (IRT) models. It can be conjectured that the selection of common items has a direct effect on the estimation error of academic abilities due to item misfit, small changes in the common items, position effect, and other sources of constructirrelevant changes between measurement occasions. Hence, the error due to the common‐item sampling could be a major source of error for the ability estimates. In operational analyses, generally two sources of error are accounted for in variance estimation: student sampling error and measurement error. A double jackknifing procedure is proposed to include a third source of the estimation error, the error due to common‐item sampling. Three different versions of the double jackknifing were implemented and compared. The data used in this study were item responses from Grade 4 students who took the NAEP 2004 and 2008 math long‐term trend (LTT) assessments. These student samples used in this study are representative samples of Grade 4 student population in 2004 and in 2008 across the US. The results showed that these three double jackknifing approaches resulted in similar standard error estimates that were slightly higher than the estimates from the traditional approach, regardless of whether an item sampling scheme was used or items were dropped at random.

Robust Haebara Linking for Many Groups: Performance in the Case of Uniform DIF

“…There is ample literature that derives standard error formulas for linking due to sampling of persons (e.g., [44,50,[63][64][65][66][67]) Alternatively, variability in estimated group means due to the selection of items has been studied as linking errors in the literature [47,[68][69][70][71][72]. In future research, it would be interesting to accompany robust Haebara linking with error components that reflect these sources of uncertainty [24,64,73]. We suppose that resampling procedures correctly reflect uncertainty due to persons and items in group mean estimates.…”

$L_p$ Loss Functions in Invariance Alignment and Haberman Linking

“…A consequence of noninvariance is that a subset of items can provide different group means even for infinite sample sizes. In large-scale assessment studies, this source of uncertainty that is due to a selection of a particular set of items has been labeled as linking errors [111][112][113][114][115][116][117][118]. Uncertainty in group means due to item sampling has also been extensively studied in generalizability theory [119][120][121].…”

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

“…Furthermore, the standard deviation for the difference in item difficulties between the two test modes was computed (DIF standard deviation, SD mode ; Camilli and Penfield, 1997). The standard errors were calculated using a double jackknife method (Xu and von Davier, 2010; see also Haberman et al, 2009), which takes into account both the uncertainty associated with the sampling of students and the uncertainty associated with items. The 39 schools were used as jackknife zones for computing the standard error associated with the sampling persons.…”