An Investigation of Methods for Reducing Sampling Error in Certain IRT Procedures

Abstract

The sampling errors of maximum likelihood estimates of item response theory parameters are studied in the case when both people and item parameters are estimated simultaneously. A check on the validity of the standard error formulas is carried out. The effect of varying sample size, test length, and the shape of the ability distribution is investigated. Finally, the ef-fect of anchor-test length on the standard error of item parameters is studied numerically for the situation, common in equating studies, when two groups of examinees each take a different test form together with the same anchor test. The results encourage the use of rectangular or bimodal ability distributions, and also the use of very short anchor tests.Until recently, the asymptotic sampling variances and covariances for maximum likelihood estimates of item parameters in item response theory (IRT) have usually been computed by assuming abilities to be known. Conversely, the asymptotic sampling variances and covariances for ability estimates have been computed by assuming the item parameters to be known. In this paper, a suggested method for computing the asymptotic sampling variance-covariance matrix of joint maximum likelihood estimates when all parameters are unknown (Lord & Wingersky, in press) is used to try to answer various practical questions. (For many purposes, an alternative approach has recently become available: the use of marginal maximum likelihood estimation, exemplified by BILOG [Mislevy & Bock, 1981], which provides asymptotic sampling variances for the estimates obtained. This approach was not available to the authors at the time the investigation reported here was initiated. It is not discussed here.) Throughout this paper all sampling variances, covariances, and standard errors are asymptotic. Section 2 presents needed additional, though not conclusive, evidence that the Lord-Wingersky method for computing the sampling variance-covariance matrix yields correct results. Section 3 investigates the effect of changing the number of items, the number of people, or the distribution of ability, on the standard errors of both the item parameters and the abilities. Section 4 presents a technique for displaying and understanding the standard errors and sampling covariances of estimates of item parameters. Section 5 deals with the situation when there are two tests that contain a set of items in common and these tests are administered to two separate groups of examinees. An important problem in item banking or test equating is to put the parameter estimates for the two tests on a common scale. One way to do this is to estimate all of the parameters for both tests in one calibration run. When this is done,

Investigating the Impact of Compromised Anchor Items on IRT Equating Under the Nonequivalent Anchor Test Design

“…The current study followed Angoff's (1984) guidelines in using 20 items as the anchor set. Previous research suggests that 20 anchor items should ensure reliable estimation of ability differences (McKinley & Reckase, 1981;Vale, Maurelli, Gialluca, Weiss, & Ree, 1981;Wingersky & Lord, 1984). The remaining 80 items were unique to each form.…”

The Changing Conception of Measurement: A Commentary

“…Therefore, in practice, interlaced designs may be less desirable than they appear. The finding that an anchor of two items is sufficient is encouraging because this is the second time that such a result has been found (see Wingersky & Lord, 1984).…”

The Examination of the Classification of Students into Performance Categories by Two Different Equating Methods

“…This is easily done using the probabilities of the IRT item characteristic functions using the algorithm developed by Wingersky and Lord (1984). Once these distributions are constructed, a Q-Q transformation provides the conditional transformation for each observed score given a specific theta estimate.…”