DIF Trees: Using Classification Trees to Detect Differential Item Functioning

Abstract

A nonparametric tree classification procedure is used to detect differential item functioning for items that are dichotomously scored. Classification trees are shown to be an alternative procedure to detect differential item functioning other than the use of traditional Mantel—Haenszel and logistic regression analysis. A nonparametric classification rule is examined through simulation and real data, and Type I error and power are compared with equivalent Mantel—Haenszel, logistic regression, and discriminant analyses.

Comparing Performances (Type I error and Power) of IRT Likelihood Ratio SIBTEST and Mantel-Haenszel Methods in the Determination of Differential Item Functioning

“…(1) in some studies, the LR method gave the same results as the MH method (Ankenmann, Witt, & Dunbar, 1996;DeMars, 2009;Vaughn & Wang, 2010), and (2) the error rate of the LR method was very high, and its statistical power, lower (Dainis, 2008;Hidalgo & Lopez-Pina, 2004;Jodoin & Gierl, 2001;Li, Brooks, & Johanson, 2012). In addition to this, the main weakness of the LR method in DIF determination is a tendency to produce higher Type I error (Li & Stout, 1996;Narayanan & Swaminathan, 1996;Rogers & Swaminathan, 1993;Swaminathan & Rogers, 1990).…”

“…The Mantel-Haenszel procedure follows a three-dimension probability table that tests the independence of two variables (Vaughn & Wang, 2010). Holland and Thayer adapted the MH procedure to detect DIF (Dorans & Holland, 1993).…”

Performances of MIMIC and Logistic Regression Procedures in Detecting DIF

“…The percentage of DIF items that were correctly detected as DIF items was calculated for power. Both Type I error and power are equally important for DIF researches (Vaughn & Wang, 2010). According to Cohen and Cohen (1983) when investigators need to set the power, it is reasonable for them to choose a value in the .70 -.90 range.…”

A Comparative Analysis of the Efficacy of Differential Item Functioning Detection for Dichotomously Scored Items among Logistic Regression, SIBTEST and Raschtree Methods

“…From the synthesis of the research related to the study of DIF for dichotomously scored items, it can be concluded that: (1) the study of the effects of DIF can be classified according to various factors, including gender, race, difficulty of the test, the distribution of the examinees' ability, the sample size, the length of the test and the number of the tests that exhibit DIF in related research, such as Shin and Wang (2009), Woods and Grimm (2011) and Demars and Lau (2011); (2) the methods commonly used to compare the efficacy of different ways of detecting DIF using CTT and IRT analytical methods are: the Mantel-Haenzel method and logistic regression, used to compare the power of the test (power rate) and type I error rate as shown in research by, amongst others, Hidalgo and Lopez-Pina (2004) and Moses, Miao, and Dorans (2010) and Vaughn and Wang (2010).…”

“…Model-based recursive partitioning is related to the classification trees method. Research by Vaughn and Wang (2010) found that the classification trees method, which detects DIF, has the power of test and type I error rate equal to the Mantel-Haenszel and logistic regression methods. Therefore, there should be a comparison of efficacy between the detection results of the new method, the Raschtree model, and the detection analysis method using classical test theory (CTT) and the analytical method using item response theory (IRT), in order to study how the efficacy of the newly-developed detection method compares with detection methods that use CTT and IRT.…”