Testing Differential Item Functioning Without Predefined Anchor Items Using Robust Regression

Abstract

Differential item functioning (DIF) occurs when the probability of endorsing an item differs across groups for individuals with the same latent trait level. The presence of DIF items may jeopardize the validity of an instrument; therefore, it is crucial to identify DIF items in routine operations of educational assessment. While DIF detection procedures based on item response theory (IRT) have been widely used, a majority of IRT-based DIF tests assume predefined anchor (i.e., DIF-free) items. Not only is this assumption strong, but violations to it may also lead to erroneous inferences, for example, an inflated Type I error rate. We propose a general framework to define the effect sizes of DIF without a priori knowledge of anchor items. In particular, we quantify DIF by item-specific residuals from a regression model fitted to the true item parameters in respective groups. Moreover, the null distribution of the proposed test statistic using robust estimator can be derived analytically or approximated numerically even when there is a mix of DIF and non-DIF items, which yields asymptotically justified statistical inference. The Type I error rate and the power performance of the proposed procedure are evaluated and compared with the conventional likelihood-ratio DIF tests in a Monte Carlo experiment. Our simulation study has shown promising results in controlling Type I error rate and power of detecting DIF items. Even when there is a mix of DIF and non-DIF items, the true and false alarm rate can be well controlled when a robust regression estimator is used.

Linking Error in the 2PL Model

“…Such a distribution is often employed in robust statistics [35]. For a prespecified DIF effect τ b , we obtain from (30) the determining equation τ = (1 To disentangle standard errors due to the sampling of persons from linking errors due to item choice, we assumed no sampling error for identified parameters {( âi1 , bi1 )} and {( âi2 , bi2 )}. That is, identified item parameters for the second group only vary across replications in the simulation study because different DIF effects f i and e i were simulated in each replication.…”

“…M-estimation theory was applied in the investigation of DIF and linking in [28][29][30]. The simultaneous treatment of standard errors and linking errors in IRT models relying on M-estimation was presented in [18,31] (see also [32]).…”

“…using a robust function that fulfills the property ρ(x)/|x| → ∞ for |x| → ∞ (see [29,30]). For example, if the median was used, ρ would be the sign function…”

Why Measurement Invariance is Not Necessary for Valid Group Comparisons

“…Alternative techniques, such as robust Haberman linking, might be preferred over invariance alignment for statistical reasons (Robitzsch, 2020). The important aspect of robust linking techniques such as invariance alignment is that some large DIF effects are treated as outliers and should be eliminated or downweighted in the estimation of group means and standard deviations (Magis & De Boeck, 2011;Robitzsch, 2022;Wang et al, 2022). The similarity of lasso-type regularization and robust linking has also been pointed out by Chen et al (2021).…”

Differential item functioning via robust scaling

“…The foregoing results provide a key idea behind this paper: The asymptotic null distribution of Y i can be obtained by using θ in place of Y i . Indeed, when comparing the R-DIF procedure to previous work on robust scaling (e.g., Stocking and Lord, 1983;He, 2013;Wang et al, 2022), the substitution in the second line of Equation ( 13) is perhaps the crucial difference. To anticipate Theorem 2 below, treating τ i as a function of θ leads directly to a proof that the R-DIF procedure has a breakdown point of 1/2.…”