Using Economic Methods Evaluatively

Abstract

As evaluators, we are often asked to determine whether policies and programs provide value for the resources invested. Addressing that question can be a quandary, and, in some cases, evaluators question whether cost-benefit analysis is fit for this purpose. With increased interest globally in social enterprise, impact investing, and social impact bonds, the search is on to find valid, credible, useful ways to determine the impact and value of social investments. This article argues that when addressing an evaluative question about an economic problem (the merit, worth, or significance of resource use), economic analysis can enhance evaluation but is usually insufficient to fully answer the evaluative question and that a stronger approach would involve explicit evaluative reasoning, supported by judicious use of economic and other methods. An overarching theoretical framework is proposed, and implications for evaluation practice are discussed.

Applying evaluative thinking to value for money: The Pakistan Sub-national Governance Programme

“…For example, by: providing a framework for systematic analysis of both costs and consequences; focusing explicitly on the opportunity cost of resource use; valuing costs and consequences in commensurable units (in the case of cost-benefit analysis); explicitly considering the relative timing of costs and consequences; and facilitating insights about risk and uncertainty through scenario and sensitivity analysis (King, 2017).…”

“…For example, cost-benefit analysis: is primarily concerned with efficiency (Adler & Posner, 2006) whereas VfM may be multi-criterial (King, 2017); takes a consequentialist perspective, whereas programme processes may have value independent from the outcomes they produce (Julnes, 2012); may obscure qualitative differences in the perspectives of different groups (e.g., power imbalances, diverging worldviews or interests) (Julnes, 2012); may in practice leave out intangible dimensions of value because they are too hard to estimate given constraints of time, evaluation resources, analyst capability or current techniques (Adler & Posner, 2006). Moreover, though normative values and assumptions from welfare economics may be widely accepted in economics and policy analysis, there could be circumstances where we do not take for granted, for example, that individuals are the best judges of their own welfare, that individuals have preferences that are stable over time, that utility increases when individual preferences are satisfied, that total utility should comprise the aggregate utilities of each individual, or that any net gain in overall efficiency (the size of the pie) is worthwhile regardless of its impact on equity (how the pie is shared) as long as the winners could potentially compensate the losers (Adler & Posner, 2006;Drummond et al, 2005).…”

“…There are multiple definitions of VfM (King, 2017). In programme evaluation, the term has tended to be used synonymously with the economic concept of efficiency (e.g., Schwandt, 2015), with economic methods of evaluation, and cost-benefit analysis in particular, widely considered the gold standard for its measurement (Julnes, 2012).…”

“…In programme evaluation, the term has tended to be used synonymously with the economic concept of efficiency (e.g., Schwandt, 2015), with economic methods of evaluation, and cost-benefit analysis in particular, widely considered the gold standard for its measurement (Julnes, 2012). Governmental, philanthropic, and international development publications have defined VfM in various ways that touch on maximising value in the procurement of inputs; delivery of outputs; achievement of outcomes; relationships between resources and outcomes (e.g., return on investment); and/or impacts on equity (King, 2017). For example, the UK's Department for International Development (DFID) defines VfM as "maximising the impact of each pound spent to improve poor people's lives" and further breaks this concept into five dimensions, labelled economy, efficiency, effectiveness, cost-effectiveness, and equity (DFID, 2011).…”

“…This article builds on six propositions from King (2017): i) that VfM is an evaluative question about an economic problem; ii) that VfM questions should be addressed through explicit evaluative reasoning; iii) that economic methods of evaluation (such as cost-benefit analysis, cost-effectiveness analysis, and cost-utility analysis) are approaches to evaluative reasoning (though perhaps not labelled as such in economic texts); iv) that economic methods of evaluation are relevant and powerful for VfM assessment; v) that economic methods of evaluation alone may be insufficient to fully answer an evaluative question about VfM; and vi) that in cases where economic methods of evaluation can enhance VfM assessment but are insufficient on their own, economic evaluation could be incorporated within a mixed methods assessment of VfM-for example, results from a cost-benefit analysis could be combined with evidence gathered using other methods (quantitative and/or qualitative), tailored to context, guided by an overarching process of evaluative reasoning. Table 1 provides further detail on these propositions.…”