Evaluating the evaluators: What have we learned from “neutral assessments” of the Canadian federal evaluation function?

Abstract

Most Canadian federal government departments and agencies are required to conduct a "neutral assessment" of their evaluation function every five years. Such assessments should focus on the capacity of the evaluation units to produce high-quality evaluation reports which serve the needs of organizational decision-makers. Our study sought to analyze neutral assessment reports produced over a ten-year period. Overall, our review suggests that the neutral assessments consistently conform with central-agency requirements as well as indicate evolving evaluation capacity and utilization. We provide recommendations for improving the function and its impact.Sommaire : La plupart des ministères et organismes du gouvernement fédéral canadien sont tenus d'effectuer un « examen neutre » de leur fonction d'évaluation tous les cinq ans. De tels examens devraient être axés sur la capacité des directions d'évaluation à produire des rapports d'évaluation de haute qualité qui répondent aux besoins des décideurs organisationnels. Le but de notre étude était d'analyser des rapports d'examens neutres effectués sur une durée de dix ans. Dans l'ensemble, notre analyse indique que les examens neutres sont invariablement conformes aux exigences des organismes centraux et indique aussi un progrès au niveau de la capacité et de l'utilisation de l'évaluation. Nous proposons des recommandations pour améliorer la fonction et ses résultats.

Translating Evaluation Policy Into Practice in Government Organizations

“…Again, the guidance offers some flexibility in terms of who can conduct the assessment, and final approval of the report rests with the Deputy Head of the organization. An in-depth description of how the neutral assessments have been conducted, based on findings from this study, can be obtained in the study by Bourgeois and Maltais (2022) .…”

“…The main methodological limit of the study was, as described earlier, that we had an incomplete set of neutral assessments, since the Heads of Evaluation submitted their reports voluntarily. In addition, although we analyzed the methodology used to conduct each neutral assessment ( Bourgeois & Maltais, 2022 ), we could not control or monitor the quality of the methodological approaches and findings presented in the neutral reports. However, we attempted to mitigate this limitation by extracting evidence and examples, rather than opinions, from the neutral assessment reports.…”