How do policy evaluators understand complexity?

Abstract

There is a well-documented interest in how insights from the study of complexity can be applied to policy evaluation. However, important questions remain as to how complexity is understood and used by policy evaluators. We present findings from semi-structured interviews with 30 UK policy evaluators working in food, energy, water and environment policy domains. We explore how they understand, use and approach complexity, and consider the implications for evaluation research and practice. Findings reveal understandings of complexity arising from contextual factors, scale-related issues and perceptions of unpredictability. The evidence indicates terminological and analogical use of complexity and its concepts by policy evaluators, but limited evidence of its literal use. Priorities for the future include framing complexity more pragmatically and as an opportunity not a cost. Communicating this up the policy hierarchy is the key to progressing complexity-appropriate evaluation – this can be enabled by strengthening links between policy evaluation and academic communities.

How does the commissioning process hinder the uptake of complexity-appropriate evaluation?

“…rhetoric and political context) and governance of a policy or programme itself which may create barriers for effective complexity-appropriate evaluation, such as too narrow focus, or a limited set of evaluation users and stakeholders. Barbrook-Johnson et al (2020) find mixed understandings and different views on the value of complexity as a lens for evaluation, among commissioners, suggesting negative views or misunderstandings (e.g. believing that being complexity-appropriate is more expensive, or creates room for uncertainty to be used as a defence by evaluators for lack of rigour, precision, or clear recommendations) may be a barrier in some cases.…”

Participatory systems mapping for complex energy policy evaluation

“…The influence of complexity science has been felt in the evaluation community for the last 20 years (Gates, 2016, Barbrook-Johnson et al 2020). Gates (2016) reviews the implications of systems thinking and complexity science for evaluation.…”

Policy evaluation for a complex world: Practical methods and reflections from the UK Centre for the Evaluation of Complexity across the Nexus

“…Still, for most, the real possibilities of envisaging how to apply complexity theory to public policy came with the ‘complexity turn’ in the social sciences in the later 1990s (Anzola et al, 2017; Barbrook-Johnson et al, 2020; Byrne, 1998; Byrne and Callaghan, 2013; Byrne and Uprichard, 2012; Urry, 2003), which followed and participated in the academy-wide excitement over the ideas coming out of the Santa Fe Institute (established 1984) and the new complexity sciences (Cairney, 2012; Gerrits, 2012; Geyer and Rihani, 2012; Gilbert et al, 2018; Haynes, 2015; Koliba and Zia, 2012; Morçöl, 2013).…”

“…Larson, 2018). Gates (2017) and Barbrook-Johnson et al (2020) also directly consider how systems and complexity approaches are understood and used by evaluators.…”