Mean Comparisons of Many Groups in the Presence of DIF: An Evaluation of Linking and Concurrent Scaling Approaches

Robitzsch, Alexander; Lüdtke, Oliver

doi:10.3102/10769986211017479

Cited by 22 publications

(33 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The presence of normally distributed random DIF is strongly different from the situation of partial invariance in which only a few items (or a few item parameters) possess DIF, while the rest of the items (or item parameters) do not have DIF. The two kinds of DIF effects can also simultaneously occur [65]. In our experience, we mainly observed random fluctuations of group-specific item parameters, which would correspond to normally distributed random DIF instead of to the case of partial invariance.…”

Section: The Role Of Normally Distributed Random Dif In Educational Assessmentmentioning

confidence: 71%

“…In this joint Haebara (HAE-joint) linking approach, distribution parameters are simultaneously estimated with common item parameters. The linking function in HAEjoint is defined as [65,93,94]:…”

Section: Haebara Linking Methods (Hae-asymm Hae-symm Hae-joint)mentioning

confidence: 99%

“…These items can be removed from linking in subsequent analysis (see, e.g., [62]). However, some research has shown that simultaneous treatment of the linking and modeling of DIF effects can result in superior statistical performance [54,[63][64][65][66].…”

Section: Random Differential Item Functioningmentioning

confidence: 99%

“…However, it seems that ensuring partial invariance is seen as a measurement ideal in educational LSA studies [67][68][69][70]. We have argued against such a perspective elsewhere [54,65,71,72] and think that the random DIF perspective with normally distributed DIF effects is more relevant in real-world applications. Moreover, we think removing items due to DIF from group comparisons is unwise because DIF can be interpreted as construct-relevant [32,52,[73][74][75].…”

Section: The Role Of Normally Distributed Random Dif In Educational Assessmentmentioning

confidence: 99%

See 3 more Smart Citations

A Comparison of Linking Methods for Two Groups for the Two-Parameter Logistic Item Response Model in the Presence and Absence of Random Differential Item Functioning

Robitzsch

2021

Foundations

Self Cite

View full text Add to dashboard Cite

This article investigates the comparison of two groups based on the two-parameter logistic item response model. It is assumed that there is random differential item functioning in item difficulties and item discriminations. The group difference is estimated using separate calibration with subsequent linking, as well as concurrent calibration. The following linking methods are compared: mean-mean linking, log-mean-mean linking, invariance alignment, Haberman linking, asymmetric and symmetric Haebara linking, different recalibration linking methods, anchored item parameters, and concurrent calibration. It is analytically shown that log-mean-mean linking and mean-mean linking provide consistent estimates if random DIF effects have zero means. The performance of the linking methods was evaluated through a simulation study. It turned out that (log-)mean-mean and Haberman linking performed best, followed by symmetric Haebara linking and a newly proposed recalibration linking method. Interestingly, linking methods frequently found in applications (i.e., asymmetric Haebara linking, recalibration linking used in a variant in current large-scale assessment studies, anchored item parameters, concurrent calibration) perform worse in the presence of random differential item functioning. In line with the previous literature, differences between linking methods turned out be negligible in the absence of random differential item functioning. The different linking methods were also applied in an empirical example that performed a linking of PISA 2006 to PISA 2009 for Austrian students. This application showed that estimated trends in the means and standard deviations depended on the chosen linking method and the employed item response model.

show abstract

Section: The Role Of Normally Distributed Random Dif In Educational Assessmentmentioning

confidence: 71%

Section: Haebara Linking Methods (Hae-asymm Hae-symm Hae-joint)mentioning

confidence: 99%

Section: Random Differential Item Functioningmentioning

confidence: 99%

Section: The Role Of Normally Distributed Random Dif In Educational Assessmentmentioning

confidence: 99%

See 2 more Smart Citations

A Comparison of Linking Methods for Two Groups for the Two-Parameter Logistic Item Response Model in the Presence and Absence of Random Differential Item Functioning

Robitzsch

2021

Foundations

Self Cite

View full text Add to dashboard Cite

show abstract

“…To enable a comparison of country means, the corresponding ability distributions can be obtained by linking approaches that establish a common ability metric [72,73]. In this article, Haberman linking [74] in its original proposal is used (see also [75,76]). The outcome of the linking procedure are country means and standard deviations.…”

Section: Linking Proceduresmentioning

confidence: 99%

On the Treatment of Missing Item Responses in Educational Large-scale Assessment Data: The Case of PISA 2018 Mathematics

Robitzsch¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Missing item responses are prevalent in educational large-scale assessment studies like the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians advocated a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. This article uses the PISA 2018 mathematics dataset to investigate the consequences of different missing data treatments on country means. Obtained country means can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

show abstract

Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons

Robitzsch

Lüdtke

2023

Structural Equation Modeling: A Multidisciplinary Journal

View full text Add to dashboard Cite

Mean Comparisons of Many Groups in the Presence of DIF: An Evaluation of Linking and Concurrent Scaling Approaches

Cited by 22 publications

References 56 publications

A Comparison of Linking Methods for Two Groups for the Two-Parameter Logistic Item Response Model in the Presence and Absence of Random Differential Item Functioning

A Comparison of Linking Methods for Two Groups for the Two-Parameter Logistic Item Response Model in the Presence and Absence of Random Differential Item Functioning

On the Treatment of Missing Item Responses in Educational Large-scale Assessment Data: The Case of PISA 2018 Mathematics

Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons

Contact Info

Product

Resources

About