No abstract

This article proposes a combination of a popular evaluation approach, contribution analysis (CA), with an emerging method for causal inference, process tracing (PT). Both are grounded in generative causality and take a probabilistic approach to the interpretation of evidence. The combined approach is tested on the evaluation of the contribution of a teaching programme to the improvement of school performance of girls, and is shown to be preferable to either CA or PT alone. The proposed procedure shows that established Bayesian principles and PT tests, based on both science and common sense, can be applied to assess the strength of qualitative and quali-quantitative observations and evidence, collected within an overarching CA framework; thus shifting the focus of impact evaluation from 'assessing impact' to 'assessing confidence' (about impact).

Commissioners of impact evaluation often place great emphasis on assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a 'contribution claim'. Current theory-based approaches fail to provide evaluators with guidance on how to collect data and assess how strongly or weakly such data support contribution claims. This article presents a rigorous quali-quantitative approach to establish the validity of contribution claims in impact evaluation, with explicit criteria to guide evaluators in data collection and in measuring confidence in their findings. Coined as 'Contribution Tracing', the approach is inspired by the principles of Process Tracing and Bayesian Updating, and attempts to make these accessible, relevant and applicable by evaluators. The Contribution Tracing approach, aided by a symbolic 'contribution trial', adds value to impact evaluation theory-based approaches by: reducing confirmation bias; improving the conceptual clarity and precision of theories of change; providing more transparency and predictability to data-collection efforts; and ultimately increasing the internal validity and credibility of evaluation findings, namely of qualitative statements. The approach is demonstrated in the impact evaluation of the Universal Health Care campaign, an advocacy campaign aimed at influencing health policy in Ghana.

This article argues that Qualitative Comparative Analysis can be a useful method in case-based evaluations for two reasons: a) it is aimed at causal inference and explanation, leading to theory development; b) it is strong on external validity and generalization, allowing for theory testing and refinement. After a brief introduction to QCA, the specific type of causality handled by QCA is discussed. QCA is shown to offer improvements over Mill's methods by handling asymmetric and multiple-conjunctural causality in addition to counterfactual reasoning. It thereby allows the explicitly separate analysis of necessity and sufficiency, recognizing the relevance of causal packages as well as single causes and of multiple causal paths leading to the same outcome (equifinality). It is argued that QCA can generalize findings to a small, medium and large number of cases.