Using network analysis to identify leverage points based on causal loop diagrams leads to false inference

Abstract

Network analysis is gaining momentum as an accepted practice to identify which factors in causal loop diagrams (CLDs)—mental models that graphically represent causal relationships between a system’s factors—are most likely to shift system-level behaviour, known as leverage points. This application of network analysis, employed to quantitatively identify leverage points without having to use computational modelling approaches that translate CLDs into sets of mathematical equations, has however not been duly reflected upon. We evaluate whether using commonly applied network analysis metrics to identify leverage points is justified, focusing on betweenness- and closeness centrality. First, we assess whether the metrics identify the same leverage points based on CLDs that represent the same system but differ in inferred causal structure—finding that they provide unreliable results. Second, we consider conflicts between assumptions underlying the metrics and CLDs. We recognise six conflicts suggesting that the metrics are not equipped to take key information captured in CLDs into account. In conclusion, using betweenness- and closeness centrality to identify leverage points based on CLDs is at best premature and at worst incorrect—possibly causing erroneous identification of leverage points. This is problematic as, in current practice, the results can inform policy recommendations. Other quantitative or qualitative approaches that better correspond with the system dynamics perspective must be explored.

Evidence-Informed Surgical Systems Strengthening with Meaningful Stakeholder Involvement in Low-Resource Settings: A Response to Recent Commentaries

“…Involving stakeholders with operational service delivery experience in network analysis and the creation of causal loop diagrams may also allow the identification of new leverage points, although a recent paper warns against erroneous cause-and-effect conclusions if no proper quantitative or qualitative approaches are applied. 10 Overall, the commentaries underscore the great potential of implementation research in support of surgical, obstetric and anaesthesia (SOA) systems strengthening, and the need for high-level policy commitment. Noteworthy developments since our paper was e-published, in August 2021, include:…”

A Systems Perspective on the Interactions Between Urban Green Infrastructure and the Built Environment

“…To clarify the question of which indicators are most connected, we count the number of connections associated with each indicator in a network analysis. However, this degree of connection needs careful interpretation, as it does not mean that highly connected indicators are more influential than others [89]. Nevertheless, the degree of connection gives an impression of the indicators for which a particularly high number of interactions and a corresponding need for further research can be expected.…”