How does the commissioning process hinder the uptake of complexity-appropriate evaluation?

Abstract

This paper investigates the role of evaluation commissioning in hindering the take-up of complexity-appropriate evaluation methods, using findings from interviews with 19 UK evaluation commissioners and contractors. We find, against a backdrop of a need to ‘do more with less’ and frustration with some traditional approaches, the commissioning process is perceived to hinder adoption of complexity-appropriate methods because of its inherent lack of time and flexibility, and assessment processes which struggle to compare methods fairly. Participants suggested a range of ways forward, including more scoping and dialogue in commissioning processes, more accommodation of uncertainty, fostering of demand from policy users, more robust business cases, and more radical overhauls of the commissioning process. Findings also emphasised the need to understand how the commissioning process interacts with the wider policy making environment and evidence culture, and how this manifests itself in different attitudes to risk in commissioning from different actors.

Why is impact measurement abandoned in practice? Evidence use in evaluation and contracting for five European Social Impact Bonds

“…Practitioners might lack the resources or expertise to commission high-quality studies (Sullivan, 2018) and government funding streams might not cover evaluation-though this has improved (Haskins and Margolis, 2015). Furthermore, actors have differing perspectives and goals (Boaz et al, 2019) and commissioners may knowingly adopt inappropriate or simplistic methods because of a lack of time and flexibility (Cox and Barbrook-Johnson, 2021) or might be unaware when they create pressure on evaluators to deviate from best practice (Pleger and Hadorn, 2018). Some argue that one solution is integrating impact estimates into contract terms (Haskins, 2018) or a combined approach where evaluation primarily has a learning purpose, but some low-level incentives are integrated into contracts (Overholser, 2018).…”

Systems thinking concepts within a collaborative program evaluation methodology: The Hermes Program Evaluation

“…Several authors have pointed out several barriers that impede the proper performance of many evaluation processes, and some of them have pointed out to the presence of political barriers and power relations that hinder effective evaluation and particularly complexity-appropriate evaluation (e.g. Barbrook-Johnson et al, 2019; Cox and Barbrook-Johnson, 2021; Schneider et al, 2016; The LSE GV314 Group, 2014; Torres-Cuello and Pinzon-Salcedo, 2022; Walton, 2016). As Haugen and Chouinard (2019) have argued, power manifests itself in a wide diversity of ways in evaluation practice, but it “.…”

Don’t panic: Bringing complexity thinking to UK Government evaluation guidance

“…These workshops also provided an opportunity to consider CECAN’s own latest research and development of complex evaluation methods, reflecting on insights from CECAN’s case studies, fellowships and events. For example, ongoing work around ways to build evaluation capacity for dealing with complexity (by one of the authors of this article) and on commissioning challenges that can inhibit the uptake of complex appropriate methods (Cox, 2019) contributed depth and detail to section ‘Key messages from the Supplementary Guide ,’ on management and commissioning. Other issues related to finding the best way to communicate complex terms in straightforward language (see section ‘Reflections’ below), and how to avoid duplication between different sections of the guidance.…”