How should Program Evaluation Standards inform the use of cost-benefit analysis in evaluation?

Abstract

Background: Cost-benefit analysis (CBA), like any other evaluation method, should be used in ways that uphold program evaluation standards and should be subjected to metaevaluation. In contrast to the broad remit of program evaluation standards, guidelines for economic evaluation focus mainly on technical aspects of evaluation quality, aimed at ensuring precision, accuracy, and reliability. Can CBA be conducted in adherence both to program evaluation standards and to its own methodological principles, or are there areas where expectations conflict? Purpose: Assess the potential for CBA to be conducted in keeping with the Program Evaluation Standards (PES) of the Joint Committee on Standards for Educational Evaluation. Setting: Analysis applies to any setting in which CBA is being considered as an evaluation method. Intervention: N/A Research Design: Methodological principles underpinning CBA were systematically assessed against the PES, to determine the extent to which CBA can be conducted in a manner aligned with these standards. CBA was rated according to whether it can follow each standard in principle, not the extent to which economists follow a given standard in practice. Data Collection and Analysis: This assessment was undertaken from a theoretical perspective, through analysis of relevant literature. The ratings are evaluative; they represent the judgments of the author, made on the basis of explicit definitions. Findings: Some ethical principles espoused in the PES are also required in CBA. On the other hand, some of the PES are not explicit requirements in CBA, though they could be applied by evaluators or economists when conducting a CBA. However, some PES logically cannot be met by CBA if it is used as a stand-alone method. All PES can theoretically be met when an evaluation combines CBA with other methods. In order to use CBA in adherence to PES, evaluators and economists must take an explicit interest in the effects of their analysis on people’s lives. This has significant implications for the way CBA should be used, including the nature and extent of stakeholder involvement, the potential use of CBA in conjunction with other methods, and decisions about when not to use CBA. As with any evaluation method, deliberation is necessary over whether, when, and how to use CBA. Keywords: cost-benefit analysis, program evaluation standards, metaevaluation

Principles and methods to advance value for money

“…Economic criteria and standards play an important role in value-for-money evaluations, but only a role because what makes resource use good for some stakeholders may not be economic in nature. It may, for example, include stewardship of resources, productive ways of working, and creating impacts with desirable qualitative characteristics like equity, sustainability, and variety (King, 2019; McLean and Gargani, 2019; Organisation for Economic Co-operation and Development (OECD), 2021) that purely economic methods may struggle to incorporate (Adler and Posner, 2006; Julnes, 2012; King, 2023).…”