Participatory systems mapping for complex energy policy evaluation

Abstract

The use of complexity science in evaluation has received growing attention over the last 20 years. We present the use of a novel complexity-appropriate method – Participatory Systems Mapping – in two real-world evaluation contexts and consider how this method can be applied more widely in evaluation. Participatory Systems Mapping involves the production of a causal map of a system by a diverse set of stakeholders. The map, once refined and validated, can be analysed and used in a variety of ways in an evaluation or in evaluation planning. The analysis approach combines network analysis with subjective information from stakeholders. We suggest Participatory Systems Mapping shows great potential to offer value to evaluators due to the unique insights it offers, the relative ease of its use, and its complementarity with existing evaluation approaches and methods.

Shifting the dynamics: implementation of locally driven, mixed-methods modelling to inform schistosomiasis control and elimination activities

“…In addition, participants were provided background on how systems maps could be used to define the parameters of simulation activities. The PSM exercises followed the process described by Barbrook-Johnson and Penn from the Centre for Evaluating Complexity Across the Nexus (CECAN) 25. After being provided with background and instructions on PSM, small groups of four to eight participants were formed based on health system level, to ease potential pressures of speaking up in the presence of workplace superiors.…”

“…to generate shared conceptual depictions of complex health issues, following on from the prominent work on 27 Systems mapping is a type of participatory modelling, used to elicit and quantify diverse perspectives from a variety of actors on causal relationships within complex systems. The process ends with a 'systems map', a diagram of explicit factors, causally linked to one another, which visually depicts a defined 'system' from the perspective of participating discussants 25 ; examples of similar outputs include causal loop diagrams and stock-and-flow diagrams. More recently, systems mapping has been embraced by some working on computational modelling and simulation methods as a way to inform models with 'real-world' information vis-a-vis the outputs of 'purposeful learning processes for action that engage the implicit and explicit knowledge of stakeholders to create formalised and shared representations of reality'.…”

“…Participatory methods have been previously used to inform health policy, practice and evaluation, most notably in relation to non-communicable diseases23 and accompanying risk factors,24 but has not been linked explicitly to modelling work and the development of evidence for decision-making in this way or context. Similar variations of this protocol used for this study have been previously described in the context of energy policy25 and ecosystem management,26 but not implemented outside of higher income countries in the context of NTDs.…”

Systems thinking concepts within a collaborative program evaluation methodology: The Hermes Program Evaluation

“…Moreover, it is frequently ignored or not comprehensively understood when it is addressed. We claim that the selection of appropriate systems thinking, systems science and complexity ideas, theories, methodologies, and methods for any evaluation should depend mainly on the purposes of the evaluation, the characteristics of the problematic situation where the evaluation takes place, and the resources and expertise available in the situation (see also Barbrook-Johnson and Penn, 2021). However, the potential presence of hidden power relations and political factors in many contexts where evaluation takes place leads us to recommend the use of up-front boundary judgments in evaluations.…”

“…Cutting-edge methods have also been presented in recent years, as well as new combinations of previously existing methods. They include the use of Participatory Systems Mapping in a couple of real-world case studies (Barbrook-Johnson and Penn, 2021), case-based modeling and scenario simulation, the presentation of a wide range of “systems mapping” methods such as Theory of Change diagrams (Wilkinson et al, 2021), as well as the use of a multimethodological approach that put together aspects and considerations from traditions such as the hard systems tradition and the neo-reductionist school of thought of complexity science (Bustamante et al, 2021).…”