Variance Estimation for Differential Test Functioning Based on Mantel‐Haenszel Statistics

Abstract

This article concerns the simultaneous assessment of DIF for a collection of test items. Rather than an average or sum in which positive and negative DIF may cancel, we propose an index that measures the variance of DIF on a test as an indicator of the degree to which different items show DIF in different directions. It is computed from standard Mantel‐Haenszel statistics (the logodds ratio and its variance error) and may be conceptually classified as a variance component or variance effect size. Evaluated by simulation under three item response models (IPL, 2PL, and 3PL), the index is shown to be an accurate estimate of the DTF generating parameter in the case of the 1PL and 2PL models with groups of equal ability. For groups of unequal ability, the index is accurate under the I PL but not the 2PL condition; however, a weighted version of the index provides improved estimates. For the 3PL condition, the DTF generating parameter is underestimated. This latter result is due in part to a mismatch in the scales of the log‐odds ratio and IRT difficulty.

Using Loss Functions for DIF Detection: An Empirical Bayes Approach

“…That is, in the formulation in (5) the across-item prior variance of the true DIF values, 'r 2, is estimated by deflating the estimated across-item variance of the DIF statistics by an amount equal to the average of the estimated item-level sampling variances. 2 Similar estimators have been independently proposed by Camilli and his colleagues (e.g., Camilli & Penfield, 1997).…”

A Generalized DIF Effect Variance Estimator for Measuring Unsigned Differential Test Functioning in Mixed Format Tests

“…This article proposes two estimators of the DIF effect variance for tests containing dichotomous and polytomous items. The proposed estimators are direct extensions of the noniterative estimators developed by Camilli and Penfield (1997) for tests composed of dichotomous items. A small simulation study is reported in which the statistical properties of the generalized variance estimators are assessed, and guidelines are proposed for interpreting values of DIF effect variance estimators.…”

“…First, measures of DTF can provide information concerning the overall impact of DIF effects when aggregated across the items of a test. For example, even if no single item is identified as displaying a large magnitude of DIF, the aggregated effects of several items with small or moderate magnitudes of DIF may be quite large (Camilli & Penfield, 1997; Rubin, 1988), suggesting that the aggregated effects may have a substantial impact on test scores. Similarly, if an item displaying a moderate or large DIF effect contains content that is critical to the test specifications (e.g., see Zieky, 1993, p. 340; Linn, 1993, p. 353), the presence of a relatively small level of DTF provides a rationale for the retention of the item because it suggests that the aggregated item‐level invariance has a negligible impact at the level of the test score.…”

Explanatory Secondary Dimension Modeling of Latent Differential Item Functioning

“…The idea of individual differences in DIF has been used earlier (Camilli & Penfield, 1997;Longford, Holland, & Thayer, 1993;Van den Noortgate & De Boeck, 2005), but it also follows, of course, from a nuisance dimension view on DIF. (b) All DIF is assumed to rely on one secondary dimension.…”