A Comparison of Differential Item Functioning Detection Methods in Cognitive Diagnostic Models

Abstract

As a class of discrete latent variable models, cognitive diagnostic models have been widely researched in education, psychology, and many other disciplines. Detecting and eliminating differential item functioning (DIF) items from cognitive diagnostic tests is of great importance for test fairness and validity. A Monte Carlo study with varying manipulated factors was carried out to investigate the performance of the Mantel-Haenszel (MH), logistic regression (LR), and Wald tests based on item-wise information, cross-product information, observed information, and sandwich-type covariance matrices (denoted by W d , W XPD , W Obs , and W Sw , respectively) for DIF detection. The results showed that (1) the W XPD and LR methods had the best performance in controlling Type I error rates among the six methods investigated in this study and (2) under the uniform DIF condition, when the item quality was high or medium, the power of W XPD , W Obs , and W Sw was comparable with or superior to that of MH and LR, but when the item quality was low, W XPD , W Obs , and W Sw were less powerful than MH and LR. Under the non-uniform DIF condition, the power of W XPD , W Obs , and W Sw was comparable with or higher than that of LR.

Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models

“…Philipp et al (2018) showed that the covariance matrix can be better estimated using an outer-product of gradient (OPG) method when all parameters are taken into consideration. Liu et al (2019) also showed that the Wald test based on the OPG method with all parameters produced better calibrated Type I error rates for the DIF detection based on the DINA model. Therefore, this study considers all parameters when calculating the covariance matrix of ϕ and V j is the submatrix related with item j .…”

“…The Wald test (Wald, 1943) is a widely used hypothesis test in statistics. In the context of CDMs, it has been used for comparing nested models (de la Torre, 2011; de la Torre & Lee, 2013; Ma & de la Torre, 2019a; Ma et al, 2016), detecting DIF (George & Robitzsch, 2014; Hou et al, 2014, 2020; Liu et al, 2019), and validating the Q-matrix empirically (Ma & de la Torre, 2019b; Terzi, 2017; Terzi & Sen, 2019). To detect DIF items using the Wald test, Hou et al (2014) calibrated data for each group separately, whereas in this study the MG-GDINA model is adopted, which calibrates multiple groups concurrently.…”

“…Hou et al (2020) have recently used the Wald test for detecting DIF under the G-DINA model. Liu et al (2019) examined the performance of the Wald test with different types of covariance matrix in DIF detection and found that the covariance matrix estimated using the complete information approach (Philipp et al, 2018) produced better calibrated Type I error than the item-wise information matrix. It should also be noted that all these studies (Hou et al, 2014; F.…”

“…Li, 2008; X. Li & Wang, 2015; Liu et al, 2019; Paulsen et al, 2020; Zhang, 2006) investigated the DIF detection based on the DINA model, which is one of the simplest CDMs and may not hold in practice. An exception is X.…”

Examining the effects of different English speech varieties on an L2 academic listening comprehension test at the item level

“…Additionally, the logistic regression test as a DIF detection method can complement the MH procedure because it enables modeling of uniform and nonuniform DIF by allowing for the interaction of the group membership and total scores (Zumbo, 2007). (See Liu et al, 2019, for comparisons among the Mantel-Haenszel, logistic regression, and Wald test methods for detecting DIF.) Uniform DIF is the presence of a systematic advantage for one group of ability-matched test takers at all levels of proficiency, whereas non-uniform DIF means that the size and direction of the gap between ability-matched groups in their probability of a correct response differs depending on the proficiency level.…”