Process Tracing and Bayesian Updating for impact evaluation

Abstract

Commissioners of impact evaluation often place great emphasis on assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a 'contribution claim'. Current theory-based approaches fail to provide evaluators with guidance on how to collect data and assess how strongly or weakly such data support contribution claims. This article presents a rigorous quali-quantitative approach to establish the validity of contribution claims in impact evaluation, with explicit criteria to guide evaluators in data collection and in measuring confidence in their findings. Coined as 'Contribution Tracing', the approach is inspired by the principles of Process Tracing and Bayesian Updating, and attempts to make these accessible, relevant and applicable by evaluators. The Contribution Tracing approach, aided by a symbolic 'contribution trial', adds value to impact evaluation theory-based approaches by: reducing confirmation bias; improving the conceptual clarity and precision of theories of change; providing more transparency and predictability to data-collection efforts; and ultimately increasing the internal validity and credibility of evaluation findings, namely of qualitative statements. The approach is demonstrated in the impact evaluation of the Universal Health Care campaign, an advocacy campaign aimed at influencing health policy in Ghana.

Evaluating outcome-based payment programmes: challenges for evidence-based policy

“…This view does not ignore individual causes of variables but examines them as 'configurations' or 'sets' in their context. (Stern et al 2009: 31) Case-based methods can be broadly typologised as either between case comparisons (such as qualitative comparative analysis) or within case analysis (the obvious example of such an approach being process tracing) (Byrne 2009, Befani andStedman-Bryce 2016). Generally, a sharp distinction between quantitative and qualitative methods is rejected (Stern et al 2012).…”

The Elusive Quest for Additionality

“…36 We cannot even be entirely certain that aid is beneficial overall, on average across countries-although the weight of the evidence points in that direction, it leaves room for doubt. 37 So the fact 34 See Collier (2011) for an introduction; Humphreys and Jacobs (2015) and Befani and Stedman-Bryce (2017) for more quantitative Bayesian applications. 35 See for example 'Civil society groups set out key recommendations ahead of OECD discussion on changes to aid rules' on the website of Eurodad, a network of civil society organizations, which asks that evidence of additionality is provided before private sector instruments can be counted as aid.…”

Applying evaluative thinking to value for money: The Pakistan Sub-national Governance Programme

“…It involves iterative analysis of causal claims along a detailed theory of change, with the insight that Bayesian inference can be applied to test evidence for causal links (Bayes, 1763;Befani & Stedman-Bryce, 2017). Overall, contribution tracing found it was possible to make a strong case that SNG activities had produced significant improvements in planning and budgeting processes but difficult to show that these improved planning and budgeting processes have yet improved service delivery.…”