Making sense of impact: A methodological framework for assessing the impact of prizes

Abstract

In recent years, debates on impact evaluation of development cooperation have flourished. There has been a marked increase in statistical impact evaluation exercises, most notably randomized experiments, and at the same time, many evaluations with a focus on impact have relied on alternative methodological approaches. The diversity in methodological approaches can be explained by differences in epistemology, characteristics of the evaluand and of the evaluation context. Broadly speaking there are two approaches: impact evaluations that rely on statistical design and/or multivariate analysis with statistical controls as a basis for attribution; and impact evaluations relying on systematic argumentation guided by some type of causal theory of change. Contribution analysis is an example of a heuristic framework that fits in the latter category and this article relates this approach to an evaluation of prizes awarded by UNESCO (The United Nations Educational, Scientific and Cultural Organization). It discusses the intervention logic underlying UNESCO prizes and the corresponding framework for empirical assessment that in part reflects the institutional context and constraints. Finally, we discuss how the evaluation can provide the basis for a full-fledged impact evaluation of a prize using contribution analysis.

Politics of policy learning: Evaluating an experiment on free pricing arrangements in Dutch dental care

“…Two distinct epistemological approaches can be identified in which learning by experimentation is discussed ( Picciotto, 2012 ; Vaessen and Raimondo, 2012 ).…”

Contribution Analysis: Theoretical and Practical Challenges and Prospects for Evaluators

“…Triangulation can involve using both qualitative and quantitative lines of evidence as required or of evidence that can include collecting existing and new evidence ( Mayne, 2012 ;Nakrošis, 2014 ). However, when faced with tight timelines and time budgets, the feasibility of using triangulation will be aff ected, thereby presenting the question of how elaborate an approach to CA should be in order to demonstrate a credible contribution claim ( Vaessen & Raimondo, 2012 ).…”

“…It is highly relevant in today's growing environment of results-based management where programs are expected to produce impacts and where evaluations of these impacts have moved away from the assessment of inputs-processes-outputs to delineating and measuring outcomes and impacts resulting from the program and how and why they occur ( Global Aff airs Canada, 2015 ; Mayne, 2004 ;Wimbush & Beeston, 2010 ). CA thus presents a heuristic framework, by which an analysis can be conducted to determine and reasonably conclude the degree of contribution based on an explicitly defi ned theory of change ( BetterEvaluation, 2013 ;Mayne, 2012 ;Vaessen & Raimondo, 2012 ).…”

“…Although considerable conceptual diversity exists with respect to theorybased evaluation approaches, Weiss (1997aWeiss ( , 1997b, one of the founders of the theory-based evaluation approach, diff erentiates among program theories, implementation theories, and theories of change. She defi nes implementation theory as an explication of how an intervention is carried out, emphasizing implementation failure and/or success based on quality, intensity, and fi delity of the intervention activities.…”

Honors: A rational choice analysis of award bestowals

“…Others are given for free but must be handed back after the recipient’s death. The low costs involved are a major reason why income and cash-constrained institutions in the non-profit and cultural sectors are bound to hand out a great variety of awards (consider, for example, the manifold prizes awarded by United Nations Educational, Scientific and Cultural Organization (UNESCO); Vaessen and Raimondo, 2012).…”