Making model judgments ROC(K)-solid: Tailored cutoffs for fit indices through simulation and ROC analysis in structural equation modeling

Abstract

Researchers commonly evaluate the fit of latent-variable models by comparing canonical fit indices (χ2, CFI, RMSEA, SRMR) against fixed cutoffs derived from simulation studies. However, the performance of fit indices varies greatly across empirical settings, and fit indices are susceptible to extraneous influences other than model misspecification. This threatens the validity of model judgments using fixed cutoffs. As a solution, methodologists have proposed four principal approaches to tailor cutoffs and the set of fit indices to the specific empirical setting at hand, which we review here. Extending this line of research, we then introduce a refined approach that allows (1) generating tailored cutoffs while also (2) identifying well-performing fit indices in the given scenario. Our so-called simulation-cum-ROC approach combines a Monte Carlo simulation with receiver operating characteristic (ROC) analysis. The Monte Carlo simulation generates distributions of fit indices under different assumptions about the population model that may have generated the data. ROC analysis helps evaluate the performance of fit indices in terms of their ability to discriminate between correctly specified and misspecified analysis models and allows selecting well-performing ones. It further identifies cutoffs for these fit indices that minimize Type I and Type II errors. The simulation-cum-ROC approach provides an alternative to fixed cutoffs, allows for more valid decisions about accepting or rejecting a model, and improves prior approaches to tailored cutoffs. We provide a shiny app that makes the application of our approach easy.

Evaluating Model Fit of Measurement Models in Confirmatory Factor Analysis

“…Hence, while tailor-made cutoffs promise to prevent misinterpretations and overly optimistic evaluations compared to fixed cutoffs that were derived for rather specific data conditions and model specifications, larger samples foster more extreme cutoffs and more likely result in model rejection, even if the actual misfit is negligible. In small samples though, the tailored ezCutoffs ( Schmalbach et al, 2019 ) seem to be too moderate as they tend to support candidate models whose fit would have been deemed not appropriate by common cutoffs (e.g., Hu & Bentler, 1999 ) or tailored cutoffs that take into account type II error rates (e.g, Groskurth et al, 2022 ; McNeish & Wolf, 2021 ). This shows that the ezCutoffs approach lacks power in small sample scenarios.…”

“…This way, a higher sample size is beneficial as the statistical power to detect meaningful deviations increases, but no discrepancies below that threshold result in rejecting the model. By evaluating the performance of the model fit indices and taking into account Type II error rates, Groskurth et al (2022) promise to assess the model fit rather independently from the actual sample size similarly to the approach of Moshagen and Erdfelder (2016) . The latter may appeal to researchers as they probably prefer choosing a critical value for close-fit (e.g., ), rather than using the ROC curve to derive a cutoff that balances Type I and Type II error rates for a given setting where an alternative model specification has to be provided as well.…”

“…The latter may appeal to researchers as they probably prefer choosing a critical value for close-fit (e.g., ), rather than using the ROC curve to derive a cutoff that balances Type I and Type II error rates for a given setting where an alternative model specification has to be provided as well. The close-fit approach of Moshagen and Erdfelder (2016) , however, relies on the known -distribution which makes it easy to calculate but limits its applicability to related indices such as the RMSEA or the GFI as Groskurth et al (2022) pointed out.…”

“…However, the ezCutoffs approach does not allow researchers to control the type-II error (as the Dynamic Fit Index Cutoffs approach does). Groskurth et al (2022) developed a different method to derive individual cutoffs tailored to the application context and empirical data at hand. Other than the two purely simulation-based approaches described earlier, Groskurth et al (2022), in a first step, repeatedly simulate data using a population model that either correspond to the actual model that should be tested (i.e., treating the hypothesized model as a correctly specified model) or to a slightly altered model that serves as a misspecified analysis model.…”

“… Groskurth et al (2022) developed a different method to derive individual cutoffs tailored to the application context and empirical data at hand. Other than the two purely simulation-based approaches described earlier, Groskurth et al (2022) , in a first step, repeatedly simulate data using a population model that either correspond to the actual model that should be tested (i.e., treating the hypothesized model as a correctly specified model) or to a slightly altered model that serves as a misspecified analysis model. 4 In a second step, the receiver–operating characteristic (ROC) curves for a set of fit indices are estimated and well-performing fit indices (i.e., indices that reach a certain performance, e.g., an area under the curve ) are selected.…”