Multilevel Factor Analysis: Reporting Guidelines and a Review of Reporting Practices

Abstract

We provide reporting guidelines for multilevel factor analysis (MFA) and use these guidelines to systematically review 72 MFA applications in journals across a range of disciplines (e.g., education, health/nursing, management, and psychology) published between 1994 and 2014. Results are organized in terms of the (a) characteristics of the MFA application (e.g., construct measured), (b) purpose (e.g., measurement validation), (c) data source (e.g., number of cases at Level 1 and Level 2), (d) statistical approach (e.g., maximum likelihood), and (e) results reported (e.g., intraclass correlations for indicators and latent variables, standardized factor loadings, fit indices). Results from this review have implications for applied researchers interested in expanding their approaches to psychometric analyses and construct validation within a multilevel framework and for methodologists using Monte Carlo methods to explore technical and methodological issues grounded in realistic research design conditions.

Is the project ‘mine’ or ‘ours’? A multilevel investigation of the effects of individual and collective psychological ownership

“…At level 2, we regressed team creativity on team engagement and the latter on both CPO and IPO, based on which we tested hypotheses 3 and 4. The fit indices for the MSEM mediation model indicated overall acceptable fit RMSEA = 0.02 CFI = 1.0, SRMR‐W = 0.03, SRMR‐B = 0.16, and comparable to the fit for multilevel models reported in the literature (Kim et al , ).…”

“…Thus, following Marsh, Hau, and Grayson () recommendations we used these as guidelines and not strict rules for each index considering the overall model fit statistics when assessing the single‐model fit. As to our knowledge there are no clear guidelines on fit indexes for multilevel models, especially for interpreting the level‐2 fit, we were also guided by recently reported ranges for each index based on a review of 58 multilevel studies (Kim, Dedrick, Cao, & Ferron, ): mean CFI = 0.95 ( SD = 0.05, min = 0.77, max = 1.0), mean RMSEA = 0.04 ( SD = 0.02, min = 0.00, max = 0.11), mean SRMR within = 0.04 ( SD = 0.02, min = 0.01, max = 0.11), mean SRMR between =.08 ( SD = 0.06, min = 0.01, max = 0.24).…”

Impact of error structure misspecification when testing measurement invariance and latent-factor mean difference using MIMIC and multiple-group confirmatory factor analysis

“…In the multilevel CFA studiesKim, Dedrick, Cao, and Ferron (2016) reviewed, the error correlations ranged from .20 to .65 (n = 22), with one outlier (.08).…”

Cross-Level Invariance in Multilevel Factor Models

“…The technical possibility to fit different models to Σ BETWEEN and Σ WITHIN , has led to applications of MLSEM where different factor structures are applied to the different levels. In a review of reporting practices of multilevel factor analyses, Kim, Dedrick, Cao, and Ferron (2016) found that 31% of the studies reported a different number of factors at the two levels. However, models with different numbers of factors at different levels are hard to interpret (Hox, Moerbeek & van der Schoot, 2017).…”

“…These types of constructs, which are frequently observed in the literature (Kim et al, 2016), are labeled "configural constructs" in a recent taxonomy by Stapleton et al (2016). Although often needed, the requirement of cross-level invariance is commonly overlooked, leading to researchers giving the same name to the factors at two levels, without actually modeling the factors accordingly.…”