Using Weighted Sum Scores to Close the Gap Between DIF Practice and Theory

Abstract

We make a distinction between the operational practice of using an observed score to assess differential item functioning (DIF) and the concept of departure from measurement invariance (DMI) that conditions on a latent variable. DMI and DIF indices of effect sizes, based on the Mantel-Haenszel test of common odds ratio, converge under restricted conditions if a simple sum score is used as the matching or conditioning variable in a DIF analysis. Based on theoretical results, we demonstrate analytically that matching on a weighted sum score can significantly reduce the difference between DIF and DMI measures over what can be achieved with a simple sum score. We also examine the utility of binning methods that could facilitate potential operational use of DIF with weighted sum scores. A real data application was included to show this feasibility.

Robustness of Weighted Differential Item Functioning (DIF) Analysis: The Case of Mantel–Haenszel DIF Statistics

“…First, we define the terms DIF and DMI used in this study, and then we illustrate MH DIF statistics with simple sum scores as the matching variable and MH DIF statistics matched with the weighted sum scores, using three different binning methods. Note that the definitions or calculations are consistent with Guo and Dorans (2019, 2020). For more discussion of these definitions, please refer to Guo and Dorans (2019, 2020).…”

“…Recently, Guo and Dorans (2019, 2020) made a distinction between DIF and DMI, in which DIF is defined as the statistics that are based on observed scores to assess items' differential performance across different groups of test takers, and DMI is defined as the departure from measurement invariance based on the measured latent ability. Guo and Dorans (2019, 2020) proposed the use of weighted sum scores (where weights are equal to the estimated discrimination parameters) in the DIF context. Through analytic methods, they showed that the DIF indices of effect sizes are different from the DMI measure under the 2PL model in many situations when a simple sum score is used as the matching or conditioning variable in DIF analysis.…”

“…To obtain weighted sum scores, item discrimination parameters are used as the item score weights, and thus the number of possible weighted sum scores can be very large, which in practice makes it infeasible to implement the weighted DIF method. Guo and Dorans (2019, 2020) proposed binning methods (e.g., naive, linear, and equipercentile) to facilitate the potential use of weighted DIF in practice.…”

“…Findings in Guo and Dorans (2019, 2020) were population results, which used model assumptions and numerical calculations without real data. A number of previous simulation studies have shown that both MH DIF and P‐DIF statistics are affected by additional factors in practice, such as test length and sample sizes (e.g., Zwick et al, 1997).…”

“…In this study, we focused on the MH DIF statistics. In the next section, we briefly review the definitions of DIF and DMI used in Guo and Dorans (2019, 2020) and this study: the calculation of MH DIF statistics matched with the simple sum scores and weighted sum scores. Then we present the simulation design, the results, and a discussion of the application of weighted DIF statistics for operational use.…”