ACT Research Report Series: A Comparison of Presmoothing and Postsmoothing Methods in Equipercentile Equating

The purpose of this study was to empirically evaluate the impact of loglinear presmoothing accuracy on equating bias and variability across chained and post-stratification equating methods, kernel and percentile-rank continuization methods, and sample sizes. The results of evaluating presmoothing on equating accuracy generally agreed with those of previous presmoothing studies, suggesting that less parameterized presmoothing models are more biased and less variable than highly parameterized presmoothing models and raw data. Estimates of standard errors of equating were most accurate when based on large sample sizes and score-level data that were not sparse. The accuracy of standard error estimates was not influenced by the correctness of the presmoothing model. The accuracy of estimates of the standard errors of equating differences was also evaluated. The study concludes with some detailed comparisons of how the kernel and traditional equipercentile continuization methods interacted with data that were presmoothed to different degrees.

Section: Discussionmentioning

confidence: 99%

Section: Presmoothing and Equating Function Accuracymentioning

confidence: 99%

Kernel and Traditional Equipercentile Equating With Degrees of Presmoothing

Moses

Holland

2007

“…RMSE(x) values so large that they are not visible in the plots correspond to score ranges of X where there is expected to be essentially no data in the samples (Figure 1). The traditional psychometric practices for addressing these issues are to smooth the raw probabilities or to average the raw probabilities with small constants (Hanson, 1990;Hanson et al, 1994;Kolen & Brennan, 2004). Pseudo Bayes methods developed in nonpsychometric contexts have been shown to estimate population probabilities more accurately than raw and modeled probabilities (Agresti, 1990;Bishop et al, 1975;Fienberg & Holland, 1973).…”

Section: Score-level Chained Equipercentile Estimation Results (Rmse(mentioning

confidence: 99%

“…In terms of psychometric practice and equipercentile equating, the choices considered for test score distribution estimation have primarily included (a) the raw probabilities, (b) the raw probabilities averaged with small constants, and (c) the smoothed probabilities based on some loglinear model (Hanson, 1990;Hanson, Zeng, & Colton, 1994;Moses & Holland, 2007 Moses and Holland (2009a) showed that pseudo Bayes estimates with w values of 0.5 improved the accuracy of equipercentile equating functions. This study broadens the Moses and Holland investigation and prior smoothing investigations by considering pseudo Bayes estimates with w values ranging between 0 and 1 along with a range of loglinear models for s .…”

Section: This Studymentioning

confidence: 99%

Pseudo Bayes Estimates for Test Score Distributions and Chained Equipercentile Equating

Moses

2009

Pseudo Bayes probability estimates are weighted averages of raw and modeled probabilities; these estimates have been studied primarily in nonpsychometric contexts. The purpose of this study was to evaluate pseudo Bayes probability estimates as applied to the estimation of psychometric test score distributions and chained equipercentile equating functions. Population test score distributions were created from actual test data and random samples of varied size were drawn from the populations. Pseudo Bayes estimation was applied to the random samples, using ranges of loglinear models and weights in the pseudo Bayes estimates' weighted averages of the raw and modeled test score probabilities. Equipercentile equating functions based on the pseudo Bayes estimates were also evaluated. Results indicated that the pseudo Bayes estimates have the potential to improve estimation accuracy for test score distributions and chained equipercentile equating functions in situations where loglinear modeling is not ideal and where finding the population loglinear model selection is not likely.

“…He also found that equating using presmoothing was about as effective as unsmoothed equating using samples twice as large. In another study, no equating and linear equating were compared to equipercentile equatings using unsmoothed, presmoothed, and postsmoothed distributions for five ACT assessment tests (Hanson, Zeng, & Colton, 1994). It was found that equipercentile equating with small samples was significantly improved when the sample distributions were smoothed, however, neither smoothing method, pre-or post-, stood out over the other.…”

Section: List Of Figuresmentioning

confidence: 99%

The Single Group With Nearly Equivalent Tests (Signet) Design for Equating Very Small Volume Multiple‐choice Tests

Grant

2011

The single group with nearly equivalent tests (SiGNET) design proposed here was developed to address the problem of equating scores on multiple‐choice test forms with very small single‐administration samples. In this design, the majority of items in each new test form consist of items from the previous form, and the new items that were administered as unscored items in the previous form. Each form is equated using data from examinees who took the previous form. As the equating is a single‐group design, with the 2 forms having a large number of overlapping items, the size of the equating sample can be much smaller than in other designs.