Lack of theory building and testing impedes progress in the factor and network literature

Abstract

The last decade has brought reforms to improve methodological practices, with the goal to increase the reliability and replicability of effects. However, explanations of effects remain scarce, and a growing chorus of scholars argues that the replicability crisis has distracted from a crisis of theory. In the same decade, the empirical literature using factor and network models has grown rapidly. I discuss three ways in which this literature falls short of theory building and testing. First, statistical and theoretical models are conflated, leading to invalid inferences such as the existence of psychological constructs based on factor models, or recommendations for clinical interventions based on network models. I demonstrate this inferential gap in a simulation study on statistical equivalence: excellent model fit does little to corroborate a theory, regardless of quality or quantity of data. Second, researchers fail to explicate theories about psychological constructs, but use implicit causal beliefs to guide inferences. These latent theories have led to problematic best practices in psychological research where inferences are drawn based on one specific causal model that is assumed, but not explicated. Third, explicated theories are often weak theories: narrative and imprecise descriptions vulnerable to hidden assumptions and unknowns. They fail to make clear predictions, and it remains unclear whether statistical effects corroborate such theories or not. Weak theories are immune to refutation or revision. I argue that these three challenges to theory building and testing are common and harmful, and impede theory formation, failure, and reform. A renewed focus on theoretical psychology and formal models offers a way forward.

How computational modeling can force theory building in psychological science

“…Failures to replicate can not only be detected but also explained and perhaps even drive theory creation as opposed to just theory rejection. Thus, building a theory explicitly as laid out in Figure 2, even if based on some hypo-and phacking, means once a phenomenon is detected we ascend the path and spend time formalising our account (e.g., Fried, 2020). Using the procedure described by our path modelthat asks for formalisation using specifications and implementations (or indeed anything more meta than an individual study, see Head et al, 2015) -"sins" out of individual scientists' control, such as questionable research practises (QRPs; see John, Loewenstein, & Prelec, 2012) committed by other labs or publication bias committed by the system as a whole, can be both discovered and controlled for in many cases.…”

Psychometric Models of Individual Differences in Reading Comprehension: A Reanalysis of Freed, Hamilton, and Long (2017)

“…The network models served not only as a reanalysis of the data and component structure assessed by Freed et al (2017), but also demonstrated how these techniques can be used to study higher-order cognitive abilities and the processes underlying them. As such, rather than proposing a common cause underlying measures of the same construct, as is the case with reflective LVMs, network theory suggests that interactions between individual processes are the driving force behind associations among parallel measures (Fried, 2020;Epskamp, Borsboom, & Fried, 2017;Epskamp & Fried, 2018). This is consistent with POT, which proposes that the positive manifold can be attributed to a sampling of shared domain-general processes (Conway & Kovacs, 2013;Kovacs & Conway, 2016).…”

“…However, the primary issue with A second issue was the choice of Freed et al (2017) to employ PCA. This analysis is not an appropriate statistical technique for making theoretical arguments or proposing theoretical solutions to a dataset (Tabachnick & Fidell, 2013) and a problematic measurement model can lead to serious issues with eventual model interpretation (Fried, 2020;Rhemtulla et al, 2019). In fact, PCA reduces the dimensionality of a dataset with no regard to the underlying latent structure of the variables (Osborne, 2014).…”

“…A tertiary goal was to present an alternative approach to examining the processes underlying cognitive abilities that is consistent with network theory as opposed to the common cause approach (Epskamp, Bosrboom &, Fried, 2017;Fried, 2020). Although the assumption is baked into the design of LVM, very few researchers explicitly indicate that they endorse the common cause perspective.…”

“…Reading comprehension and other higher-order cognitive abilities (i.e., abilities that require multiple lower-order cognitive processes) are typically examined using LVM, because these analyses offer a means of isolating and measuring the specific contributions of each latent construct. When Spearman (1904) developed reflective factor analysis from cognitive test data, it was under the assumption that there was a common latent cause underlying performance on similar cognitive tasks (the common cause; Pearl, 2000; see also Fried, 2020;Schmittmann et al, 2013). In other words, scores on two similar measures are correlated because both assess the same underlying construct that is directly causing the level of performance that is obtained on the individual measures.…”