An Investigation of Methods for Reducing Sampling Error in Certain Irt Procedures*

Abstract

The sampling errors of maximum likelihood estimates of item‐response theory parameters are studied in the case where both people and item parameters are estimated simultaneously. A check on the validity of the standard error formulas is carried out. The effect of varying sample size, test length, and the shape of the ability distribution is investigated. Finally, the effect of anchor‐test length on the standard error of item parameters is studied numerically for the situation, common in equating studies, where two groups of examinees each take a different test form together with the same anchor test. The results encourage the use of rectangular or bimodal ability distributions, also the use of very short anchor tests.

Optimum Design in Item Response Theory: Test Assembly and Item Calibration

“…Obviously, in this type of equating an optimality problem is involved, as there is a trade-off between the errors in the parameter estimates for the two item sets and in the estimate of the linking equation. Several studies of "blocking", "spiraling" , and "interlacing" designs have been conducted to empirically assess the quality of these types of designs (see, e.g., Davey, 1992;Vale, 1986;Wingersky and Lord, 1984), but straightforward optimization has not yet been attempted.…”

Optimal sequential designs for on-line item estimation

“…A D-optimal design maximizes the determinant of the Fisher information matrix; and, an A-optimal design minimizes the trace of the inverse of the Fisher information matrix (Federov, 1972;and Silvey, 1980). The criteria used in Wingersky and Lord (1984) and Stocking (1990) are related to the theory of A-optimality.…”

“…In estimating these item parameters, this means roughly that for 100 observations from the normal design, one may obtain the same amount of accuracy with 12 observations at -0.82 and 12 observations at 0.47. That the normal design performs better than the rectangular design raises a question about the consistency of these results with those of and Lord (1984) and Stocking (1990). An explanation is that we are comparing designs according t(; the criterion of D-optimality whereas they compare designs by a criterion directly related to A-…”

Optimal On‐Line Calibration of Testlets

“…If the population of abilities is normal, then random matching amounts to sampling from a normal population. Recent research has shown that the precision of parameter estimates can be significantly increased from distributions other than the normal distribution (Jones and Jin, 1994;Stocking, 1990;Wingersky and Lord, 1984). Even in very large samples from a normal population, very little information may be available for calibrating some items, and that the success of a particular item calibration using item response theory depends heavily on the selection of more informative data (Stocking, 1990).…”