Permutation randomization methods for testing measurement equivalence and detecting differential item functioning in multiple-group confirmatory factor analysis.

Abstract

In multigroup factor analysis, different levels of measurement invariance are accepted as tenable when researchers observe a nonsignificant (Δ)χ2 test after imposing certain equality constraints across groups. Large samples yield high power to detect negligible misspecifications, so many researchers prefer alternative fit indices (AFIs). Fixed cutoffs have been proposed for evaluating the effect of invariance constraints on change in AFIs (e.g., Chen, 2007; Cheung & Rensvold, 2002; Meade, Johnson, & Braddy, 2008). We demonstrate that all of these cutoffs have inconsistent Type I error rates. As a solution, we propose replacing χ2 and fixed AFI cutoffs with permutation tests. Randomly permuting group assignment results in average between-groups differences of zero, so iterative permutation yields an empirical distribution of any fit measure under the null hypothesis of invariance across groups. Our simulations show that the permutation test of configural invariance controls Type I error rates better than χ2 or AFIs when the model contains parsimony error (i.e., negligible misspecification) but the factor structure is equivalent across groups (i.e., the null hypothesis is true). For testing metric and scalar invariance, Δχ2 and permutation yield similar power and nominal Type I error rates, whereas ΔAFIs yield inflated errors in smaller samples. Permuting the maximum modification index among equality constraints control familywise Type I error rates when testing multiple indicators for lack of invariance, but provide similar power as using a Bonferroni adjustment. An applied example and syntax for software are provided. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Teachers' Conceptions of Assessment: A Global Phenomenon or a Global Localism

“…It is also worth noting that the standards used in this study to evaluate equivalence in multi-group comparisons rely on conventional standards and approaches (Byrne, 2004). More recently, the use of permutation tests has been proposed as a superior method for testing metric and scalar invariance, because permutations can control Type I error rates better than the conventional approaches used in this study (Jorgensen et al, 2018;Kite et al, 2018). Combined with determination of effect sizes for metric and scalar equivalence tests (dMACS) (Nye and Drasgow, 2011), these are methodological approaches that may lead future analyses to identification of more universal results.…”

Testing Measurement Invariance with Ordinal Missing Data: A Comparison of Estimators and Missing Data Techniques

“…Another limitation is that we assume that researchers have the correctly specified configural invariance model. Testing configural invariance could be challenging because its 2 statistic can be rejected for one of the following two reasons, or both: (1) the factor structure is not identical across groups, and (2) model misspecification is not relevant to measurement invariance (Jorgensen, Kite, Chen, & Short, 2018). It would be worthwhile to examine whether model misspecification could affect the relative performance of the strategies.…”

Measurement memo I: Updated practices in psychological measurement for sexual scientists

“…Ideally, accessible tools will eventually follow to help researchers precisely calibrate their models to reasonable fit index cut-offs. Although new analytic techniques are already available to assist in this calibration (see Jorgensen et al, 2018), they likely require a degree of sophistication beyond what the typical everyday user is prepared to indulge. Sexual scientists therefore wishing to revise their analytic practice to take model reliability into account when evaluating model fit might consider calculating average or median standardized loadings, and using McNeish et al's Table 1 to derive rough cut-offs that may be more reasonable, until more accessible and comprehensive tools emerge.…”

“…Complicating matters further, the subfield of basic science in psychological measurement modeling is a living, breathing area of scholarship. New and improved techniques are continuously developed and disseminated to provide researchers with increasingly sophisticated and rigorous tools for extracting psychological entities inside a person's head and representing them in a valid numeric form that a researcher can use (e.g., Jorgensen, Kite, Chen, & Short, 2018). Interdisciplinary and multidisciplinary fields that do not occasionally check in to survey these new methodological developments are therefore at risk of being on the outside looking in for best practices in psychological measurement.…”