A Systematic Review of Meta-Evaluations: Lessons for Evaluation and Impact Analysis

Abstract

Impact evaluation and measurement are highly complex and can pose challenges for both social impact providers and funders. Measuring the impact of social interventions requires the continuous exploration and improvement of evaluation approaches and tools. This article explores the available evidence on meta-evaluation—the “evaluation of evaluations”—as an analytical tool for improving impact evaluation and analysis in practice. It presents a systematic review of 15 meta-evaluations with an impact evaluation/analysis component. These studies, taken from both the scholarly and gray literature, were analyzed thematically, yielding insights about the potential contribution of meta-evaluation in improving the methodological rigor of impact evaluation and organizational learning among practitioners. To conclude, we suggest that meta-evaluation is a viable way of examining impact evaluations used in the broader social sector, particularly market-based social interventions.

Reviewing Michael Scriven’s Contributions to JMDE

“…viiviii) Over half a century later, his argument still rings true: Metaevaluation has matured over the decades from a concept to a pillar among researchers, practitioners, and institutions that rely on the now-essential tool for improving practice and supporting decisions (Jacobs & Affodegon, 2015). It was added as a critical aspect of the Joint Committee on Standards for Educational Evaluation's program evaluation standards (Yarbrough et al, 2010) and remains a flexible method when examining the strengths, weaknesses, and utility of published evaluations (Qian-Khoo, 2022).…”

Emotional impact of online assessment strategies in engineering mathematics during COVID-19 confinement: A study in Mexico City

“…Assessing students is a key part of education, as noted by Wanzer (2021). Qian-Khoo et al (2022) described this assessment as a continuous effort to track students' progress in learning and skill development through their involvement in teaching and learning activities. Educational evaluation involves gathering evidence to measure and show a student's level of learning and to decide if they have successfully met the requirements for a particular subject, according to Wang and Yu (2021).…”