Client-centered meta-evaluation

Abstract

Persons who commission evaluation studies often do so because the context and information requirements for their decisions are very complex. Decision makers face practical and political constraints. They frequently make decisions based on competing information sources and they often need to adjudicate among differing interests regarding program processes and outcomes. Cronbach and his associates (1980) discuss this decision setting as a context of accommodation in contrast to a context of command. Their description of accommodation focuses on political andj or vested interests of individual stakeholders regarding particular program policy or action. As a result of accommodation, Cronbach and associates suggest that decisions are rarely made; rather, they emerge. This point of view has implications for evaluators. For example, their "25th Thesis" states: "In a context of accommodation, the evaluator cannot expect a 'gojno-go' decision to turn on his assessment of outcomes"; further, their "Thesis 26" states: "What is needed is information that supports negotiation rather than information calculated to point out the 'correct' decision" (p. 4).An anchoring premise for the present article is that evaluators, like decision makers, operate in a context of accommodation. In a typical evaluation study, the evaluator frequently must adjust, or accommodate, his or her activities to conform to a budget, timeline, or other factors associated with the object of evaluation. The belief that evaluation occurs in a context of accommodation suggests changes in our traditional conception of meta-evaluation. We believe that such changes will make meta-evaluation more useful to clients.We propose a definition of meta-evaluation that emphasizes two functions: One function is to help evaluators improve their practice, and the other is to help clients make better use of evaluation information.AUTHORS' NOTE: We wish to thank Dan Stufflebeam for his review of early drafts of this manuscript.

Meta-evaluation: Validating program evaluation standards through the United Nations Evaluation Quality Assessment (EQAs)

“…The inclusion of meta-evaluation in the Joint Committee on Standards for Educational Evaluation’s Program Standards (Third Edition, 2010) cements the importance of including meta-evaluation in every evaluation. In fact, meta-evaluation has various forms, including summative (Scriven, 1969), client-centred (Reineke & Welch, 1986), concurrent (Hanssen et al, 2008), responsive (Sturges & Howley, 2017), and internal formative meta-evaluations (Harnar et al, 2020). Early work in the development of meta-evaluation recognised the challenge of establishing criteria focused on technical quality criteria, such as reliable measures, valid conclusions, and cost-effectiveness, while remaining flexible enough for stakeholder needs (Apthorpe & Gasper, 1982; Smith, 1982).…”

Meta-avaliação de auditorias de natureza operacional do tribunal de contas da união

“…Para fins de compreensão esclarecemos que há dois tipos de meta-avaliação: a centrada na avaliação e a centrada no cliente. A primeira é detalhada nesta pesquisa e a segunda pode ser descrita, brevemente, como forma de aproximar os resultados produzidos pela avaliação, ao uso pelos clientes das avaliações (Reineke & Welch, 1986); conforme Woodside & Sakay (2001), está mais relacionada ao desempenho, impacto do programa e tomada de decisão do gestor.…”