Dynamic Fit Index Cutoffs for Confirmatory Factor Analysis Models

Abstract

<p>Model fit assessment is a central component of evaluating confirmatory factor analysis models. Fit indices like RMSEA, SRMR, and CFI remain popular and researchers often judge fit based on suggestions from Hu and Bentler (1999), who derived cutoffs that distinguish between fit index distributions of true and misspecified models. However, methodological studies note that the location and variability of fit index distributions – and, consequently, cutoffs distinguishing between true and misspecified fit index distributions – are not fixed but vary as a complex interaction of model characteristics like sample size, factor reliability, number of items, and number of factors. Many studies over the last 15 years have cautioned against fixed cutoffs and the faulty conclusions they can trigger. However, practical alternatives are absent, so fixed cutoffs have remained the status quo despite their shortcomings. Criticism of fixed cutoffs stem primarily from the fact that they were derived from one specific confirmatory factor analysis model and lack generalizability. To address this, we propose dynamic cutoffs such that derivation of cutoffs is adaptively tailored to the specific model and data being evaluated. This creates customized cutoffs that are designed to distinguish between true and misspecified fit index distributions in the researcher’s particular context. Importantly, we show that the method does not require knowledge of the “true” model to accomplish this. As with fixed cutoffs, the procedure requires Monte Carlo simulation, so we provide an open-source, web-based Shiny application that automates the entire process to make the method as accessible as possible.</p>

Why We Need to Abandon Fixed Cutoffs for Goodness-of-Fit Indices: An Extensive Simulation and Possible Solutions

“…We believe this insight should be internalized by all (applied) researchers and included in statistics and methods curricula. Moreover, while we understand that our findings may sound pessimistic and leave some readers wondering how to approach model evaluation in the future, we are convinced that the fundamental issues with GOFs that we (and others before us; e.g., Marsh, 2004;McNeish & Wolf, 2020) identified have a silver lining: They can encourage researchers to think more deeply about the appropriateness of global cutoff heuristics for GOFs and explore alternative procedures that will ultimately lead to more valid judgments of model fit. Below, we outline several promising avenues.…”

“…Two other strategies to arrive at tailored cutoffs are superior to the rather simplistic first strategy. They can be broadly summarized as regression-/equation-based (Moshagen & Erdfelder, 2016;Nye & Drasgow, 2011) or simulation-based approaches (McNeish & Wolf, 2020;Millsap, 2007Millsap, , 2013Niemand & Mai, 2018;Schmalbach et al, 2019;Pornprasertmanit, 2014;Pornprasertmanit et al, 2020).…”

“…In stark contrast to the Hu and Bentler (1999) cutoffs, results of these simulation studies appear to have been largely ignored in applied research. The same goes for cautionary notes regarding overgeneralizations of fixed cutoffs (e.g., Heene et al, 2011;Markland, 2007;Marsh et al, 2004;McNeish & Wolf, 2020;Niemand & Mai, 2018;Nye & Drasgow, 2011).…”

Measurement Invariance Testing Using Confirmatory Factor Analysis and Alignment Optimization: A Tutorial for Transparent Analysis Planning and Reporting

“…Indeed, recent research suggests the use of dynamic fit index cut-offs that are computed based on the characteristics of the examined factor model and not universally fixed (McNeish & Wolf, 2021). Moreover, equivalence testing approaches with multi-group structural equation modeling have demonstrated some evidence of superior performance to both the chi-square test and fixed fit index approaches with respect to error control, but may require greater sample sizes to achieve adequate statistical power (Yuan & Chan, 2016;Counsell et al, 2020).…”

Affective structure, measurement invariance, and reliability across different experience sampling protocols

“…This applies for instance to cut-offs for fit indices. Recent work has highlighted that these cut-offs are highly dependent on characteristics of the model (McNeish & Wolf, 2020). However, simulation work for multilevel CFAs is sparse.…”