Evaluating outcome-based payment programmes: challenges for evidence-based policy

Abstract

We review the state of evaluation within outcome-based commissioning in the United Kingdom. This is the first review to include empirical evaluations of both PbR and SIB programmes. We find a paucity of evaluation and that the quality of evaluations is not high. Moreover, studies tend to conflate the outcomes-based commissioning mechanism with the intervention or services that are funded, and are unable to assess the contribution of these separate elements to impact. Our review also highlights the challenges faced by evaluators in measuring social outcomes. We suggest ways to address these challenges.

Backing what works? Social Impact Bonds and evidence-informed policy and practice

“…It may be the case that the commercial sensitivities of investors militate against the commissioning of independent evaluation (Warner, 2013). The opportunity for in-depth evaluation makes the SIB concept of particular interest to the academic community and evaluation specialists (Fox & Morris, 2019). There are opportunities to combine (quantitative) impact evaluations with (qualitative) process evaluations and cost-effectiveness studies of SIBs, thereby furthering knowledge of 'what works, why, when and for whom' (Pawson, Tilley, & Tilley, 1997)delivering research which transcends traditional 'hierarchies of evidence' of effectiveness and closer to a comprehensive approach to evaluation (Boaz et al, 2019) that includes qualitative as well as quantitative data.…”

“…. The opportunities that SIB-financed interventions offer for formal evaluation (Fox & Morris, 2019).…”

Widening perspectives on social impact bonds

“…However, in practice, public services contracting is complex and there is limited evidence in support of paying entirely on outcomes from the performance-pay literature (Lagarde et al 2013). Technical challenges also exist in evaluation which makes it difficult to attribute changes in outcomes to the SIB itself (McHugh et al 2013;Sinclair et al 2014;Roy, McHugh, and Sinclair 2017) and in the absence of independent evaluation with a counterfactual, attribution relies on observed outcomes rather than comparison against a control group (Fox 2019, Fraser et al 2018a). SIBs therefore merit critical examination to assess whether, and how, they might deliver on their promises.…”

“…The available empirical evidence from the UK is comprised of largely qualitative evaluations with no rigorous attempt to test outcomes against a counterfactual control or comparator group or to demonstrate attribution so that interventions are paid-out based on observed qualitative outcomes control or comparator group (Mason, Lloyd, and Nash 2017;Thomas, Griffiths, and Pemberton 2014;Disley et al 2015Disley et al , 2011Fraser et al 2018a;Anders and Dorsett 2017). See Fox (2019) of this special issue for a systematic review of the empirical evidence from the UK. This undermines the basic claim of SIBsthat they enable governments to only pay for demonstrated outcomes.…”

“…This was the first attempt to collect academic perspectives on SIBs with the aim of critically exploring and challenging dominant practitioner narratives of the SIB as a "win-win" solution for governments and service providers. This special issue brings together a number of theoretical developments (Morley 2019, Sinclair, McHugh, andRoy 2019) that consider the moral implications of SIBs and critique the financialisation of public services and service users, as well as highlight the need for evaluation (Fox 2019) and more direct empirical analysis of SIBs (Carter 2019, Fraser et al 2019. These papers provide a valuable contribution to the nascent theoretical and empirical literature on SIBs to challenge the "win-win" narrative that SIBs can enable accountable and efficient public services that achieve blended social benefits alongside private financial returns.…”