Focusing Narrowly on Model Fit in Factor Analysis Can Mask Construct Heterogeneity and Model Misspecification: Applied Demonstrations across Sample and Assessment Types

Abstract

This study builds upon research indicating that focusing narrowly on model fit when evaluating factor analytic models can lead to problematic inferences regarding the nature of item sets, as well as how models should be applied to inform measure development and validation. To advance research in this area, we present concrete examples relevant to researchers in clinical, personality, and related subfields highlighting two specific scenarios when an overreliance on model fit may be problematic. Specifically, we present data analytic examples showing that focusing narrowly on model fit may lead to (a) incorrect conclusions that heterogeneous item sets reflect narrower homogeneous constructs and (b) the retention of potentially problematic items when developing assessment measures. We use both interview data from adult outpatients (N = 2,149) and self-report data from adults recruited online (N = 547) to demonstrate the importance of these issues across sample types and assessment methods. Following demonstrations with these data, we make recommendations focusing on how other model characteristics (e.g., factor loading patterns; carefully considering the content and nature of factor indicators) should be considered in addition to information provided by model fit indices when evaluating factor analytic models.

Constructs derived from the addiction cycle predict alcohol use disorder treatment outcomes and recovery 3 years following treatment.

“…Models were estimated separately for each study using a random split-half sample, including potentially cross-loading terms via target rotation, and iteratively testing all possible indicators of the addiction cycle domains. The number of indicator variables was reduced in the first split-half sample based on factor loadings, model fit statistics, comparative fit index (CFI) > .96; root-mean-square error of approximation (RMSEA) < .06; standardized root mean square (SRMR) < .08, coefficient omega (Ω) reliability, and substantive considerations (Stanton et al, 2022). The final model was then replicated in the remaining random half of the data, and model fit was excellent, COMBINE: χ 2 (33) = 41.20, p = .15; CFI = .998; RMSEA = .019; SRMR = .017; MATCH: χ 2 (25) = 70.88 p < .001; CFI = .982; RMSEA = .046; SRMR = .025.…”

Locating triarchic model constructs in the hierarchical structure of a comprehensive trait-based psychopathy measure: Implications for research and clinical assessment.

“…The important implication of this demonstration is that evidence derived from factor-analytic modeling alone is inadequate for determining the “correct” number of dimensions underlying a set of measured attributes. As has been highlighted elsewhere (e.g., Montoya & Edwards, 2021; Patrick et al, 2021; Stanton et al, 2023), relying on internal model fit statistics to select the number of factors to extract from a given inventory is fraught with limitations, and the current work serves to illustrate how an understanding of the hierarchical unfolding of construct indicators can be informed by examination of pertinent external correlates.…”