Finally! A Valid Test of Configural Invariance Using Permutation in Multigroup CFA

Abstract

In multigroup factor analysis, configural measurement invariance is accepted as tenable when researchers either (a) fail to reject the null hypothesis of exact fit using a ¦ 2 test or (b) conclude that a model fits approximately well enough, according to one or more alternative fit indices (AFIs). These criteria fail for two reasons. First, the test of perfect fit confounds model fit with group equivalence, so rejecting the null hypothesis of perfect fit does not imply that the null hypothesis of configural invariance should be rejected. Second, treating common rules of thumb as critical values for judging approximate fit yields inconsistent results across conditions because fixed cutoffs ignore sampling variability of AFIs. As a solution, we propose replacing ¦ 2 and fixed AFI cutoffs with permutation tests. Iterative permutation of group assignment yields an empirical distribution of any fit measure under the null hypothesis of invariance. Simulations show the permutation test of configural invariance controls Type I error rates better than ¦ 2 or AFIs when a model has parsimony error (i.e., negligible misspecification) but the factor structure is equivalent across groups (i.e., the null hypothesis is true).

Applying Permutation Tests and Multivariate Modification Indices to Configurally Invariant Models That Need Respecification

“…Because the LRT is a test of overall exact fit of the model to the data, two potential sources of misspecification are confounded (Cudeck and Henly, 1991 ; MacCallum, 2003 ): estimation discrepancy (due to sampling error) and approximation discrepancy (due to a lack of correspondence between the population and analysis models). Because configural invariance is assessed by testing the absolute fit of the configural model, the LRT for a multigroup model further confounds two sources of approximation discrepancy (Jorgensen et al, 2017 , in press ): the overall discrepancy between population and analysis models could be partitioned into (a) differences between groups' true population models and (b) discrepancies between each group's population and analysis models. The H 0 of configural invariance only concerns the former source of approximation discrepancy (which I will refer to as group discrepancy ), whereas the latter source is an issue of model-fit in general (which I will refer to as overall approximation discrepancy ).…”

“…Evaluating overall fit therefore tests the wrong H 0 by confounding group equivalence and overall exact model fit into a single test. The permutation method introduced by Jorgensen et al ( 2017 , in press ) disentangles group discrepancy from overall approximation discrepancy.…”

“…The LRT appears to be used as the sole criterion to evaluate ME/I only half as often (16.7%) as (Δ)AFI(s) alone (34.1%), the most popular of which is the comparative fit index (CFI; Bentler, 1990 ), at least in the context of ME/I (Putnick and Bornstein, 2016 ). Given the sensitivity of (Δ)AFI sampling distributions to data and model characteristics (Marsh et al, 2004 ), basing conclusions about configural invariance on AFIs (e.g., interpreting CFI >0.95 as evidence of good approximate fit) leads to Type II errors in large samples, but can also lead to inflated Type I errors in small samples (Jorgensen et al, 2017 ). Permutation also provides a solution to problems with unknown (Δ)AFI sampling distributions by comparing observed configural-model AFIs to empirical sampling distributions derived under the H 0 of equivalent group configurations (Jorgensen et al, in press ).…”

“…However, the null hypothesis (H 0 ) of configural invariance is distinct from the H 0 of overall model fit. Permutation tests of configural invariance yield nominal Type I error rates even when a model does not fit perfectly (Jorgensen et al, 2017 , in press ). When the configural model requires modification, lack of evidence against configural invariance implies that researchers should reconsider their model's structure simultaneously across all groups.…”

“…Using a newly proposed permutation test of configural invariance (Jorgensen et al, 2017 , in press ), the H 0 of configural invariance can be tested with nominal Type I error rates even when the H 0 of correct specification is false. I extend this line of research by proposing the use of multivariate modification indices (Bentler and Chou, 1992 ) to guide researchers in respecifying their inadequately fitting configural models when there is no evidence against the H 0 of group equivalence in true model configurations.…”