Statistical Assessment of Estimated Transformations in Observed‐Score Equating

Abstract

Equating methods make use of an appropriate transformation function to map the scores of one test form into the scale of another so that scores are comparable and can be used interchangeably. The equating literature shows that the ways of judging the success of an equating (i.e., the score transformation) might differ depending on the adopted framework. Rather than targeting different parts of the equating process and aiming to evaluate the process from different aspects, this article views the equating transformation as a standard statistical estimator and discusses how this estimator should be assessed in an equating framework. For the kernel equating framework, a numerical illustration shows the potentials of viewing the equating transformation as a statistical estimator as opposed to assessing it using equatingspecific criteria. A discussion on how this approach can be used to compare other equating estimators from different frameworks is also included. 106

A Comparison of IRT Observed Score Kernel Equating and Several Equating Methods

“…They found that IRTKE offered small standard errors and biases under most circumstances. Further, Wiberg (2016) investigated how ability changes between two test administrations affected the IRTKE and other equating methods in NEAT. Lacking of true equating criterion in empirical data, they did not draw much conclusions about which method was better performed.…”

“…Lacking of true equating criterion in empirical data, they did not draw much conclusions about which method was better performed. Meanwhile, researchers put forward some new methods by combing KE with other methods, such as the local IRTKE, local KE (Wiberg et al, 2014), and linear IRTKE (Wiberg, 2016). To sum up, the newly proposed IRTKE has been theoretically validated for its superiority to other methods, but few simulated studies are carried out to verify its equating performances when compared with the CTT methods (such as EE) and IRT methods (such as IRTOSE), which is one of major objectives in this study.…”

“…So, the IRT simulation, indeed, helps. However, there is always another concern about the possible unfairness to certain equating methods caused by the IRT, itself (Harris and Crouse, 1993;Godfrey, 2007;Choi, 2009;Norman Dvorak, 2009;Wiberg and González, 2016;Andersson and Wiberg, 2017;Kim et al, 2017;De Ayala et al, 2018). In detail, a simulation study backgrounded on the IRT may be partial to some relevant equating methods, such as IRTOSE and IRTTSE, and disadvantage others.…”

“…EE, IRT, KE, or IRTKE? One function might favor equating results under a similar theoretical framework (Qu, 2007;Ricker and von Davier, 2007;Choi, 2009;Chen, 2012;Wiberg and González, 2016). That is, the criteria calculated by the EE reference might lead the EE, KE, even IRTKE to smaller errors compared with the IRT, as these methods are exactly EE, itself, or its extension.…”

Kernel Equating Using Propensity Scores for Nonequivalent Groups

“…The three estimated equating transformations were evaluated in terms of bias, root mean squared error (RMSE) and standard error (SE), as defined in Wiberg & González (2016)…”