How Robust Is the p Factor? Using Multitrait-Multimethod Modeling to Inform the Meaning of General Factors of Youth Psychopathology

Abstract

We used multitrait-multimethod (MTMM) modeling to examine general factors of psychopathology in three samples of youths ( Ns = 2,119, 303, and 592) for whom three informants reported on the youth’s psychopathology (e.g., child, parent, teacher). Empirical support for the p-factor diminished in multi-informant models compared with mono-informant models: The correlation between externalizing and internalizing factors decreased, and the general factor in bifactor models essentially reflected externalizing. Widely used MTMM-informed approaches for modeling multi-informant data cannot distinguish between competing interpretations of the patterns of effects we observed, including that the p factor reflects, in part, evaluative consistency bias or that psychopathology manifests differently across contexts (e.g., home vs. school). Ultimately, support for the p factor may be stronger in mono-informant designs, although it does not entirely vanish in multi-informant models. Instead, the general factor of psychopathology in any given mono-informant model likely reflects a complex mix of variances, some substantive and some methodological.

Conceptual, methodological, and measurement factors that disqualify use of measurement invariance techniques to detect informant discrepancies in youth mental health assessments

“…In youth mental health, Converging Operations ( Figure 3A ) manifests when informants’ reports yield data that produce the same research findings (e.g., both reports support a youth client’s anxiety diagnosis, both reports support significant reductions in symptoms post-treatment). In fact, within Converging Operations anything less than converging findings reflects a measurement confound (e.g., random error, rater biases; Watts et al, 2022 ).…”

“…Second, they would have to conduct studies testing for the presence of rater biases in informant discrepancies that correct for the limitations we noted in prior work on the depression-distortion hypothesis (see Online Supplementary Material). Future work on the depression-distortion hypothesis would have to contend with the reality that the proposed source of measurement confounds in this theory (i.e., informants’ levels of depression) contain, at minimum, a “mix” of domain-relevant variance and variance reflecting measurement confounds (see also Watts et al, 2022 ). However, all rater bias studies, to date, have made no attempt to deconstruct the variance in the purported source of rater biases (e.g., parent’s levels of depressive symptoms) into domain-relevant and domain- irrelevant components.…”

“…Measurement invariance techniques are not alone in emphasizing common variance. The majority of analytic techniques currently used by researchers seeking to integrate or model multi-informant data in youth mental health emphasize common variance and treat unique variance as error, including composite scores (e.g., Martel et al, 2021 ), many applications of structural equations modeling (e.g., Eid et al, 2008 ; Bauer et al, 2013 ; Howe et al, 2019 ; Watts et al, 2022 ), and combinational algorithms (e.g., AND rule; Baumgartner et al, 2021 ).…”

Integrating multi-informant reports of youth mental health: A construct validation test of Kraemer and colleagues’ (2003) Satellite Model

“…Conversely, variance unique to any one source (i.e., as reflected by informant discrepancies) reflects error, and thus a core assumption underlying use of composite scoring is that estimates of psychological phenomena should emphasize shared or common variance among reports (see Edgeworth, 1888;Borsboom, 2005). This same rationale underlies applications of structural equation modeling (SEM) to integrating or modeling multi-informant data (see also Eid et al, 2008;Watts et al, 2021), combinational algorithms used to integrate data collected within diagnostic interviews (i.e., AND/OR rules; see Offord et al, 1996;Rubio-Stipec et al, 2003;Youngstrom et al, 2003;Valo and Tannock, 2010), and recent applications of measurement invariance techniques to detect informant discrepancies (e.g., Russell et al, 2016;Olino et al, 2018;Murray et al, 2021;Florean et al, 2022).…”

“…For instance, integrative approaches that emphasize both common variance and domain-relevant unique variance outperform composite scores in terms of magnitudes of relations to criterion variables ( De Los Reyes et al, 2022b ) and direct tests of incremental validity ( Makol et al, 2020 ). Further, the most commonly used MTMM-informed structural models cannot distinguish between informant discrepancies that reflect measurement confounds from those that reflect domain-relevant information ( Watts et al, 2021 ). This work points to the need for guidance on approaches to integrating multi-informant data in youth mental health.…”