Exact Versus Asymptotic Mantel‐Haenszel DIF Statistics: A Comparison of Performance Under Small‐Sample Conditions

Abstract

This study evaluated exact testing (Agresti, 1992) as a method for conducting Mantel-Haenszel DIF analyses (Holland & Thayer, 1988) with relatively small samples. Sample-size restrictions limit the standard asymptotic Mantel-Haenszel for many practical applications; however, new developments in computing technology have made exact testing procedures feasible. The highly discrete distributions that are likely to occur in small-sample DIF analyses could yield very different results for asymptotic versus exact methods. It is therefore important to determine under controlled conditions the extent to which the exact approach is effective in correctly identifying DIF. A series of computer simulations were conducted in which 3 levels of induced bias (IRT b-parameter differences between groups of.25, .50, and .75) and 4 sample sizes (reference group = 500, focal group = 25, 50, 100, and 200) were investigated. Power comparisons at .01 and .05 alpha levels were carried out between the exact testing procedure and the conventional Mantel-Haenszel.

A Review of Recent Developments in Differential Item Functioning

“…CMH (Meyer et al, 2004;Parshall & Miller, 1995) has been studied as a potential solution and may be worthy of further investigation, because comparisons of its statistical and practical significance findings to those of exact nonparametric approaches produced comparable results. Other approaches to small sample DIF analysis, for example, Bayesian methods (Sinharay et al, in press) or smoothing techniques (Puhan et al, 2007), have not succeeded in lowering the minimum sample size threshold.…”

“…The first MH extension used exact, instead of asymptotic, statistics (Parshall & Miller, 1995) to analyze DIF for small samples. Another extension aimed at reducing Type I error and increasing power used MH methods for multiple subgroup comparisons (Penfield, 2001).…”

Exact tests for the rasch model via sequential importance sampling

“…Small numbers of examinees overall, or small numbers of examinees in one of the subgroups have been shown to cause problems for the Mantel-Haenszel test of item bias (Pashall & Miller, 1995;Mazor, Clauser & Hambleton, 1992;Shepard, Camilli & Williams, 1985). The Mantel-Haenszel test is based on a large sample normal approximation to the distribution of the test statistic.…”

A Review of Ets Differential Item Functioning Assessment Procedures: Flagging Rules, Minimum Sample Size Requirements, and Criterion Refinement

“…63). Parshall and Miller (1995) compared the MH chi-square (without continuity correction) to a procedure based on the exact permutation test. The reference group sample size was 500 and the focal group sample size ranged from 25 to 200.…”