Complex Versus Simple Modeling for DIF Detection

Abstract

Previous research has demonstrated that differential item functioning (DIF) methods that do not account for multilevel data structure could result in too frequent rejection of the null hypothesis (i.e., no DIF) when the intraclass correlation coefficient (ρ) of the studied item was the same as the ρ of the total score. The current study extended previous research by comparing the performance of DIF methods when ρ of the studied item was less than ρ of the total score, a condition that may be observed with considerable frequency in practice. The performance of two simple and frequently used DIF methods that do not account for multilevel data structure, the Mantel–Haenszel test (MH) and logistic regression (LR), was compared with the performance of a complex and less frequently used DIF method that does account for multilevel data structure, hierarchical logistic regression (HLR). Simulation indicated that HLR and LR performed equivalently in terms of significance tests under most conditions, and MH was conservative across most of the conditions. Effect size estimate of HLR was equally accurate and consistent as effect size estimates of LR and MH under the Rasch model and was more accurate and consistent than LR and MH effect size estimates under the two-parameter item response theory model. The results of the current study provide evidence to help researchers further understand the comparative performance between complex and simple modeling for DIF detection under multilevel data structure.

DIF Analysis with Multilevel Data: A Simulation Study Using the Latent Variable Approach

“…In DIF analysis, the standard statistical test logistic regression (LR) and Mantel-Haenszel test (MH) have been shown to be not as effective as hierarchical logistic regression (HLR) in controlling type I error rate because LR and MH do not model random effects due to clusters (French & Finch, 2010, 2013. Jin et al (2014) extended French and Finch's study by showing that HLR outperformed LR when the intraclass correlation (ρ) was medium to large (e.g., ρ > 0.25), but LR performed equally well as HLR when ρ was small to medium (e.g., ρ < 0.25). Their findings were based on the sufficiency of the covariate (i.e., the total score), which, by convention, is called the matching variable in DIF analysis.…”

“…The significance test of the regression coefficient γ 1 in Equation (1) is used to determine the presence of uniform DIF. X i is the covariate (i.e., the total score) to When data were sampled from clusters, HLR was recommended for DIF analysis by researchers to account for dependency between person level scores (French & Finch, 2010, 2013Jin et al, 2014), especially when ρ was medium to large. The HLR model is written as, (2) Where, G j = 1 for focal group and G j = 0 for reference group, X ij is the person level covariate (i.e., the total score), and the random components .…”

“…According to Jin et al (2014), LR and HLR performed equivalently when ρ was small to medium with no MAD, given that the person level covariate was a sufficient estimate of θ for both reference and focal groups. The authors of the current study hypothesized that with short tests and the presence of MAD, both DIF methods, especially LR, will be more likely to fail to control type I error rate at the nominal level, because the presence of MAD between groups might be falsely classified as DIF, especially when the reliability of the covariate is low due to fewer items in a test.…”

“…Tests with 5 and 10 items were generated to represent short tests as compared to previous studies with long tests (e.g., 20-item test, Jin et al, 2014). The number of items selected was within the range of short tests defined in previous research (e.g., Woods, 2009).…”

“…A variety of differential item functioning (DIF) methods have been developed to test for MI. Recently, the performance of several DIF methods have been examined and compared under the multilevel data structure, where the local independence assumption was violated (French & Finch, 2010, 2013Jin, Myers, & Ahn, 2014). Results of these studies have shown that standard DIF methods that did not account for multilevel data structure may or may not perform equivalently as modified DIF methods that accounted for multilevel data structure under specific conditions.…”