Applications of contribution analysis to outcome planning and impact evaluation

Abstract

Contribution analysis is a structured approach to theory-based impact evaluation originally developed in Canada in the context of Results-Based Management (RBM) although there have been few examples of contribution analysis in practice since Mayne's original paper (2001). We argue that contribution analysis adds value to other theory-based evaluation approaches by providing a more structured and rigorous approach to participatory evaluation planning, data analysis and reporting. It can be applied in the context of participatory strategic planning and performance monitoring as well as impact evaluation. Examples are drawn from Scotland and Canada in the performance context of RBM in Canada and Outcomes-Based Accountability (OBA) in Scotland. The authors argue that, as a participatory process, contribution analysis strengthens both conceptual and practical understanding of planning/managing for outcomes and implementation and change theories, thus helping to build collaborative capacity within and across partner organizations. For public managers, the contribution analysis process has a strong appeal and practical value when faced with the task of demonstrating the contribution of single organizations to addressing complex social issues while working in partnership with other public agencies facing multiple accountabilities.

How practitioner-led research could have greater impact: the importance of considering knowledge mobilisation holistically

“…This collaboration is one of the more valued aspects of the contribution analysis methodology (Wimbush, Montague and Mulherin, 2012). Coupled with robust evidencegathering, we have found that this reflective approach to evaluation can also generate new theoretical understandings of the change process.…”

Contribution Analysis: Theoretical and Practical Challenges and Prospects for Evaluators

“…In such situations, CA can provide an attractive alternative evaluation approach as it does not require the use of counterfactual-based evaluation designs nor promises unequivocal understandings of the direct cause-and-eff ect relationships between an intervention and the observed results ( Befani & Mayne, 2014 ;Dybdal, Nielsen, Lemire, & Ramboll Management Consulting, 2011 ;Mayne, 2008 ). By seeking to provide evidence by which one can "reasonably" assert the contribution of an intervention to observed results, it relies on tests of alternative explanations to act as substitute candidates in place of counterfactuals to determine the plausibility of a proposed theory of change and the signifi cance and, accordingly, the infl uence of the intervention in producing observed changes ( Befani & Mayne, 2014 ;Lemire, 2010 ;Wimbush, Montague, & Mulherin, 2012 ).…”

“…Due to these advantages, CA has been increasingly applied in areas such as the fi eld of international development evaluation where intended, and actual causal relationships between interventions and results are heavily dependent on the situational context, including the considerable amount of time that may be required before results are observed ( Kotvojs & Shrimpton, 2007 ;Noltze, Gaisbauer, Schwedersky, & Krapp, 2014 ). Other examples of CA in practice include various government-driven interventions with complicated or complex program characteristics (including causal chains) and undefi ned time frames for expected results ( Biggs, Farrell, Lawrence, & Johnson, 2014 ;Delahais & Toulemonde, 2012 ;Wimbush, Montague, & Mulherin, 2012 ). With increasing requirements from government and funding agencies to move away from evaluations based solely on implementation performance metrics and toward measurable impacts, CA is intended to provide evaluators with the ability to conclude with a reasonable degree of confi dence that an intervention is contributing to observed results and to provide an evidence-based justifi cation of how this is probably occurring ( Kotvojs & Shrimpton, 2007 ).…”

“…Th e primary goals of the evaluation were to "determine whether there was a reasonable theory-of-change driving the program" ( Biggs, Farrell, Lawrence, & Johnson, 2014, p. 216) and also "whether the program was contributing to the intended goal" (p. 216) using the CA approach. Mayne's original steps were confi gured based on the evaluation requirements and program context and augmented with tools from more recently published literature advancing the approach ( Dybdal, Nielsen, Lemire, & Ramboll Management Consulting, 2011 ;Kotvojs & Shrimpton, 2007 ;Lemire, Nielsen, & Dybdal, 2012 ;Wimbush, Montague, & Mulherin, 2012 ). We will present this evaluation as an exemplar of the CA evaluation approach, followed by a discussion of other potential areas of clarifi cation and improvement.…”

Assessing the Practice Impact of Research on Evaluation

“…Using CA relies on ensuring that the ToC is sufficiently complex to reflect the many intricacies that may arise later in the evaluation process while avoiding overcomplicating the framework to the point where the evaluation becomes unfeasible (Delahais & Toulemonde, ). Building a strong ToC requires ensuring that alternate explanations have been properly identified (Delahais & Toulemonde, ; Mayne, ; Wimbush, Montague, & Mulherin, ). In their experience, Delahais and Toulemonde () maintain that researchers pay greater attention to drawing causal links between the boxes in the ToC than to exploring alternative explanations.…”