Quantifying the Reliability and Replicability of Psychopathology Network Characteristics

Abstract

Acknowledgements: Thanks to Nicholas R. Eaton and Eiko I. Fried for comments on an earlier version of this manuscript. Thanks also to Fried and colleagues-who authored the study on the replicability of posttraumatic stress disorder symptom networks (Fried et al., 2018, Clin Psychol Sci)-for making their summary data, output, and code available, and for encouraging reanalysis of the data for further replicability research. AbstractPairwise Markov random field networks-including Gaussian graphical models (GGMs) andIsing models-have become the "state-of-the-art" method for psychopathology network analyses. Recent research has focused on the reliability and replicability of these networks. In the present study, we compared the existing suite of methods for maximizing and quantifying the stability and consistency of PMRF networks (i.e., lasso regularisation, plus the bootnet and NetworkComparisonTest packages in R) with a set of metrics for directly comparing the detailed network characteristics interpreted in the literature (e.g., the presence, absence, sign, and strength of each individual edge). We compared GGMs of depression and anxiety symptoms in two waves of data from an observational study (n = 403) and reanalyzed four posttraumatic stress disorder GGMs from a recent study of network replicability. Taken on face value, the existing suite of methods indicated that overall the network edges were stable, interpretable, and consistent between networks, but the direct metrics of replication indicated that this was not the case (e.g., 39-49% of the edges in each network were unreplicated across the pairwise comparisons). We discuss reasons for these apparently contradictory results (e.g., relying on global summary statistics versus examining the detailed characteristics interpreted in the literature), and conclude that the limited reliability of the detailed characteristics of networks observed here is likely to be common in practice, but overlooked by current methods. Poor replicability underpins our concern surrounding the use of these methods, given that generalizable conclusions are fundamental to the utility of their results.Keywords: network analysis; network theory of mental disorders; psychopathology networks; conditional independence; replicability crisis Quantifying the reliability and replicability of psychopathology network characteristicsThe central tenet of the network theory of mental disorders is that psychopathology is due to dynamic causal interactions among symptoms (Borsboom, 2017). This theory encourages the quantification of the symptom-level structure of psychopathology using network analysis-a method with rapidly growing popularity in psychopathology research.

The network approach to psychopathology: promise versus reality

“…While it may be naive to expect exact replication, observed levels of inconsistency between networks seem particularly problematic in the context of a theory that emphasizes interpretation of the presence, absence, strength and sign of each individual edge and the corresponding centrality of individual symptoms. Importantly, optimistic perspectives on the reliability and replicability of symptom networks are often based on methods (e.g., bootnet, the omnibus NetworkComparisonTest, and correlations between lists of edges) that shift the focus away from these detailed features, and towards global network patterns that do not correspond with the basis of network theory or the insights that symptom networks have been promoted to provide. The result is that these popular methods create an impression of reliability and replicability that fails to translate to the level at which networks are interpreted.…”

Learning to Live with Sampling Variability: Expected Replicability in Partial Correlation Networks

“…Partial vs. zero-order correlations. In Forbes et al (2019), the above model was suggested to have inherent limitations that contribute to unreliable findings compared to, say, zero-order correlations ξ ij = F (r ij ) ("association networks"). The key difference is that zero-order correlations are not conditioned on any variables and partial correlations are conditioned on k − 2 variables.…”

“…The key difference is that zero-order correlations are not conditioned on any variables and partial correlations are conditioned on k − 2 variables. Although not mentioned in Forbes et al (2019), controlling for variables does present two underappreciated hurdles for evaluating replicability:…”

The replicability and generalizability of internalizing symptom networks across five samples.

“…Some have argued that network interpretations and clinical implications focus on the presence, sign, and strength of specific edges and the rankorder of node centrality (i.e., which symptom is most central, second most central, third most central, etc.). An examination of these characteristics within and across two samples of Major Depressive Disorder (MDD) and Generalized Anxiety Disorder (GAD) symptoms revealed that 83.4-86.6% of edges replicated within and between networks but only 16.7-55.6% of individual nodes' centrality rank-order matched within and between samples (Forbes et al, 2017a), leading to the conclusion that "popular network analysis methods produce unreliable results" (Forbes et al, 2017a; see also Forbes, Wright, Markon, & Krueger, 2019). 2 2 Steinley, Hoffman, Brusco, and Sher (2017) provided a commentary on Forbes et al's (2017a) paper in which they introduced a new method termed fixed-margin sampling for examining whether network model results differ from what would be expected by chance.…”