A cautionary note on testing latent variable models

Abstract

The article tackles the practice of testing latent variable models. The analysis covered recently published studies from 11 psychology journals varying in orientation and impact. Seventy-five studies that matched the criterion of applying some of the latent modeling techniques were reviewed. Results indicate the presence of a general tendency to ignore the model test (χ2) followed by the acceptance of approximate fit hypothesis without detailed model examination yielding relevant empirical evidence. Due to reduced sensitivity of such a procedure to confront theory with data, there is an almost invariable tendency to accept the theoretical model. This absence of model test consequences, manifested in frequently unsubstantiated neglect of evidence speaking against the model, thus implies the perilous question of whether such empirical testing of latent structures (the way it is widely applied) makes sense at all.

Explaining the Ability to Learn Analogies: The Role of Executive Functions and Fluid Intelligence

“…The analysis of the covariance matrix was conducted using the maximum likelihood estimation method in AMOS 22.0 (Arbuckle, 2013). Given an expected adequate statistical power (computed in R) with regard to the specific model complexity and sample size, a significant χ 2 value (p < .05) was regarded a sufficient criterion for model rejection, irrespective of the approximate goodness-of-fit indices (see Hayduk, 2014;Ropovik, 2015). For a non-rejected model, the following approximate indices were further examined: the Comparative Fit Index (CFI), the Tucker-Lewis Index (TLI), the Root Mean Square Error of Approximation (RMSEA), the Standardized Root Mean Square Residual (SRMR) and the Bayesian Information Criterion (BIC).…”

Best Practice Recommendations for Using Structural Equation Modelling in Psychological Research

“…Overall model fit, which includes the chi-square test, tests precisely what it describes: whether the model fits the observed data. Ropovik (2015) notes that, while a statistically significant chi-square value is often ignored on the grounds that the test itself is overly sensitive when large samples are used, the "only message that a significant χ 2 tells is… take a good look at that model [as] something may be wrong here" (p. 4). Further, the attainment of fit using other indices (e.g., GFI or RMSEA) does not necessarily mean that the chi-square test was statistically significant because of a trivial misspecification.…”

Assessing the validity of a learning analytics expectation instrument: A multinational study

“…Although the alternative fit indices were suggestive of an acceptable fit, the Χ 2 test was found to be significant; thus, an inspection of local fit was warranted (Kline, ; Ropovik, ). Starting with the modification indices and standardized expected parameter change values, there were two possible modifications to be made by freely estimating the correlated errors between Items 7 and 8 (MI = 10.19, SEPC = 0.37) and Items 11 and 12 (MI = 18.47, SEPC = 0.61).…”

“…On the basis of alternative fit indices, the two‐factor model could be regarded as having an acceptable fit, but an assessment of local fit was required due to the significant Χ 2 test (Kline, ; Ropovik, ). There were only two absolute residual correlation values ≥.10 (Table ), which were between Items 2 and 5 (−.10) and Items 11 and 12 (.14).…”