Making rigorous causal claims in a real-life context: Has research contributed to sustainable forest management?

Abstract

This article reflects on an evaluation commissioned by the Centre for International Forestry Research, an international research centre working on tropical forests. In the Congo basin, it took from 10 to 20 years for research works to influence the sustainability of forest management through a complex web of interactions between timber companies, national governments, international organizations, development agencies, NGOs, and consultancies. By applying the contribution analysis approach, the evaluation was able to trace several causal pathways which percolated through this web of interactions and resulted in a number of contributions always indirect and marginal but sometimes necessary. The article discusses how contribution claims were inferred from evidence, what the underlying logic of causal arguments was, how some contributions could be qualified as necessary ones, and how far the evaluation went on the way to generalization. The discussion bridges contribution analysis with process tracing and realist evaluation. Keywords contribution analysis, inferring causality, mechanism, process tracing, realist evaluation, research evaluation, theory of change This article begins with a real-life example of contribution analysis (CA) and then reflects on how evaluators may make credible causal claims where the theory of change is complex. CA

Using Actor-Based Theories of Change to Conduct Robust Evaluation in Complex Settings

“…As noted, ToCs are used in many theory-based evaluation approaches, such as realist evaluation and contribution analysis. Contribution analysis (CA) (Mayne, 2001(Mayne, , 2009(Mayne, , 2011(Mayne, , 2012a) is increasingly being used in evaluations of interventions in order to address the perennial challenge of "dealing with causality, " particularly in complex systems where changes in outcomes are the result of a number of factors in addition to the intervention-that is, situations where there are several factors acting together to bring about change, with each factor being a contributory cause and none being sufficient on its own to bring about the desired change (Mayne, 2012a; see also Buckley, 2016;Buregeya, Brousselle, Nour, & Loignon, 2017;Delahais & Toulemonde, 2017;Downes, Novicki, & Howard, 2018;Kane, Levine, Orians, & Reinelt, 2017;Mayne, 2012b;Noltze, Gaisbauer, Schwedersky, & Krapp, 2014;Terrapon-Pfaff, Gröne, Dienst, & Ortiz, 2018;and Ton, 2017. The aim is to make credible causal claims about the contribution an intervention is making to observed results. CA argues that if one can verify or confirm a theory of change with empirical evidence, 1 then it is reasonable to conclude that the intervention in question has made a difference.…”

Making and justifying evidence claims: evidence synthesis of impact evaluations and systematic reviews in international development

“…The process also involves combining judgements with defined concepts of 'quality' and 'value'. Delahais and Toulemonde (2017) reflect on how causal contribution claims were made based on an evaluation related to sustainable forest management. They generalise this by considering the credibility of causal claims where a theory of change is complex.…”

Revisiting Contribution Analysis

“…As such, it explores how and why interventions are working and for whom. Contribu tion analysis is increasingly being used in evaluations (Buckley, 2016;Buregeya, Brousselle, Nour, & Loignon, 2017;Delahais & Toulemonde, 2017;Downes, Novicki, & Howard, 2018;Kane, Levine, Orians, & Reinelt, 2017;Noltze, Gais bauer, Schwedersky, & Krapp, 2014;Ton, 2017), and in particular to address causal issues in complex settings (Koleros & Mayne, 2019;Palladium, 2015).…”