Using Loss Functions for DIF Detection: An Empirical Bayes Approach

Abstract

We investigated a DIF flaggbzg method based on loss functions. The approach builds on earlier research that involved the development of an empirical Bayes (EB) enhancement to Mantel-Haenszel (MH) DIF analysis. The posterior distribution of DIF parameters was estimated and used to obtain the posterior expected loss for the proposed approach and for competing classification rules.Under reasonable assumptions about the relative seriousness of Type I and Type II errors, the loss-function-based DIF detection rule was found to perform better than the comnzonly used "A, " "B, " and "C" DIF classification system, especially in small samples.

Empirical Bayes Versus Standard Mantel-Haenszel Statistics for Detecting Differential Item Functioning Under Small Sample Conditions

“…We show the item parameters used in the simulation in Table 1; the same as those used by Zwick et al (1999Zwick et al ( , 2000. For all 34 items of the study, we set c i parameters at .15.…”

“…The rule we employ, taking into account the considerations mentioned by Zwick et al (2000), establishes that if V P D i + (M P D i ) 2 > 1, the item will be classified as a DIF item.…”

“…In theory, the first advantage derived from the use of Bayesian methods in DIF detection is that they should offer more stable results than does the standard MH procedure. For example, Zwick et al (2000) found that the use of a loss function (LF) on the basis of Bayesian statistics for detecting DIF gave greater power than did the ordinary MH procedure in sample sizes of 200 and 50 examined in the reference and focal groups, respectively. Zwick and Thayer obtained the same result when they applied this procedure to a computerized adaptive test (CAT) in which the samples were extremely small.…”

Contributions to the Quantitative Assessment of Item, Test, and Score Fairness

“…The FB Bayesian DIF analysis method was shown to be an improvement over existing methods in a simulation study. Zwick et al (2000) proposed a Bayesian updating (BU) method that may avert the shrinkage associated with the EB and FB approaches. Zwick et al (2012) implemented the BU approach and compared it to the EB and FB approaches in both simulated and empirical data.…”

“…The EB approach also conveyed information about DIF stability in a more useful way by representing the state of knowledge about an item's DIF status as probabilistic. Zwick et al (2000) investigated a DIF flagging method based on loss functions. The approach built on their earlier research that involved the development of an EB enhancement to MH DIF analysis.…”