Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation

Abstract

This article proposes a combination of a popular evaluation approach, contribution analysis (CA), with an emerging method for causal inference, process tracing (PT). Both are grounded in generative causality and take a probabilistic approach to the interpretation of evidence. The combined approach is tested on the evaluation of the contribution of a teaching programme to the improvement of school performance of girls, and is shown to be preferable to either CA or PT alone. The proposed procedure shows that established Bayesian principles and PT tests, based on both science and common sense, can be applied to assess the strength of qualitative and quali-quantitative observations and evidence, collected within an overarching CA framework; thus shifting the focus of impact evaluation from 'assessing impact' to 'assessing confidence' (about impact).

Integrating theory-based evaluation and process tracing in the evaluation of civil society gender budget initiatives

“…Certainty tests -hoop tests -disconfirm hypotheses, as they enable the ruling out of some mechanisms if the evidence is not found (Befani and Mayne, 2014). When multiple independent hoop tests are combined to test a hypothesis, the result is 'an additive effect that increases our confidence in the validity of (h) [the hypothesis] given that the probability of nonvalid hypotheses surviving multiple independent hoop tests fails after each successive hoop' (Beach and Pedersen, 2013: 105).…”

“…With the increasing complexity of real-world interventions, counterfactual designs have been found to be inappropriate (Befani and Mayne, 2014;Stern et al, 2012) and recent years have seen the rise of non-counterfactual designs being applied in 'small-n' studies to evaluate the effects of complex interventions (White and Phillips, 2012). White and Phillips (2012: 7) identify four types of evaluation approaches that draw on implicit theories 'to establish beyond reasonable doubt how an outcome or set of outcomes occurred':…”

Blended Finance Evaluation

“…23 In light of the various blending instruments and the incentives they provide for private sector involvement, this is obviously an important question. The main arguments for public intervention are that there are additional social returns beyond the expected private returns or that the private sector overstates the risks or underestimates the private returns of the interventions (Barder, O. and Talbot, 2015 [32]). As Barder and Talbot note, there is no case for blending (or public intervention more in general) if the combination of risks and returns is competitive to alternative investment opportunities.…”

“…30 The focus on ESG standards and the SDGs seems only gradually to be incorporated in the interventions, 31 and there were few examples of assessments of tradeoffs between financial returns, ESG considerations and development effects. 32 Finally, some evaluations, but not all, assessed the level of increased tax payments in developing countries as a result of the interventions. 33 Evaluations varied in the extent to which they included assessments of the justification for the selection and design of specific interventions.…”

Applying evaluative thinking to value for money: The Pakistan Sub-national Governance Programme

“…Additionally, other development programmes may contribute to the intended outcomes. In this context, and without the possibility of a measurable counterfactual, a theory-based approach called contribution tracing (Befani & Mayne, 2014) was used to assess the strength of causal claims.…”