The Contribution of Metaevaluation to Program Evaluation: Proposition of a Model

Abstract

Background: This theoretical article points to the fundamental difference between meta-analysis and metaevaluation. A model of metaevaluation for social programs is presented based on prior practical research. Purpose: The purpose is to present a model for metaevaluation as a tool that can be used in other studies. Theory points to the need of a qualitative framework to go beyond the understanding of meta-analysis for program evaluation. Setting: This theoretical article is based on an empirical research conducted at a Brazilian Governmental audit agency. Subjects: The Government agency where the practical research was conducted is responsible for the effectiveness and accountability of social programs through audits that occurred from 2003 to 2006. Intervention: Meetings and interviews were held with auditors that participated in the evaluation process going from planning to final reports as the model proposes. Research Design: The model for metaevaluation has a qualitative approach used to evaluate prior evaluations for social programs. Data Collection and Analysis: Data collection included structured interview with the chief manager of the agency in charge of evaluating governmental programs. Documents and reports were analyzed using qualitative method for content analysis. Synthesis of categories was applied to compare different analysis and summarize findings. Findings: Metaevaluation and meta-analysis are different research methods with a different approach. Meta evaluation is a qualitative method useful when evaluating prior evaluations. Yet the quantitative approach of meta-analysis applies better for first evaluations. Meta evaluation may include other methods to help strengthen the evaluation results. Conclusions: Metaevaluation aligns theory and practice for program evaluation. The proposed model for metaevaluation may hold value for future theoretical and empirical work.

Development, Implementation, and Meta-Evaluation of a National Approach to Programmatic Assessment in Canadian Family Medicine Residency Training

“…Evaluation data for CRAFT come from multiple sources, making a meta-evaluation 28 an appropriate approach. Using a theory-led, outcomes-focused program evaluation framework, our meta-evaluation focused on the assumptions of CRAFT using 2 sources of data: an institutional audit of assessment information and processes in 4 programs that were early adopters of CRAFT (Table 2), and published evaluation data (Appendix 1).…”