Overview of Evaluation Methods for R&D Programs. A Directory of Evaluation Methods Relevant to Technology Development Programs

Abstract

This booklet provides a quick reference guide to evaluation methods for R&D managers in the U.S. Department of Energy's Technology Development programs. 2 While peer review is the form of R&D evaluation most frequently used by R&D managers, there are other evaluation methods which are also useful-particularly for estimating program outcomes and impacts retrospectively. This booklet provides an overview of 14 evaluation methods that have proven useful to R&D program managers in Federal agencies. Each method is briefly defined, its uses are explained, its limitations are listed, examples of its successful use by other R&D managers are provided, and references are given.

Agricultural research impact assessment

“…A wide variety of methods for RIA are available. Based on the assessment of the US Advanced Technology Programprobably the most important RIA initiative ever - Ruegg and Feller (2003:17) The list presented by Ruegg and Jordan (2007) in their overview of evaluation methods for R&D programmes is even more comprehensive and includes: econometrics, mission/impact mapping, foresighting, etc. Based on a survey of US public agencies, most of the organisations involved in R&D perform some kind of 'case studies'.…”

“…Case studies are also widely used to explore topics for which no strong theory exists. For Ruegg and Jordan (2007), another strength of case studies is their ability to put flesh on the bones of quantitative approaches, i.e. to provide elements of contexts and qualitative explanation about quantitative trends.…”

Deep impact: re-conceptualising university research impact using human cultural accumulation theory

“…Measures, therefore, may need to include a component of user engagement with research outputs by counting the number of views, shares, saves or even media coverage an output receives (Marcella et al 2015), the documentation needed to determine the effect of engagement on cultural learning, for example through backward and forward process tracing (Ruegg and Jordan 2007). It may well be, however, that since data may need to be collected as a project unfolds, process tracing may be the most intensive, but thorough, way to measure cultural change and any resultant residual impact, since process tracing requires documentation of all project interactions (Beach and Pederson 2013).…”

ASIRPA: A comprehensive theory-based approach to assessing the societal impacts of a research organization

“…All the approaches mentioned above rely on case studies, a very frequent method for RIA (Ruegg & Jordan 2007). Despite their potential pitfalls (lack of objectivity, quantification, reproducibility), case studies are favored because they offer engagement 'with complexity [and offer] a detailed, in-depth understandings (…) about the description of events or initiatives over which the researcher has little or no control' (Bell et al 2011, p. 228).…”