Evaluations as a decent knowledge base? Describing and explaining the quality of the European Commission’s ex-post legislative evaluations

Abstract

Evaluations are a potentially important tool for democratic governments: they provide a basis for accountability and policy learning. To contribute to these key functions, evaluations must be of sufficient methodological quality. However, this quality is threatened by both political influences and technical complexities. This article describes and explains the variance in the quality of ex-post legislative (EPL) evaluations conducted by the European Commission, which is a frontrunner in this realm. A number of potential political and technical explanations of evaluation quality are tested with a unique, self-constructed dataset of 153 EPL evaluations. The results show that the Commission's EPL evaluations usually apply a robust methodology, while the clarity of their scope, the accuracy of their data and the foundations of their conclusions are problematic. The variance in this quality is mainly explained by the type of evaluator: EPL evaluations conducted by external actors are of higher quality than evaluations conducted internally by the Commission.

Explaining the political use of evaluation in international organizations

“…While these findings do not claim that evaluation reports as such are biased, we demonstrate that their political use differs systematically. These findings concur with others who argue that the idea of evidence-based policy-making should not be taken for granted (Cooley and Snyder 2015;Hinterleitner et al 2016;Højlund 2015;Merry 2011;Porter 1995;van Voorst and Mastenbroek 2019). We do not suggest to refrain from using evaluation in public management, but the revealed politics surrounding seemingly functional tools like evaluation should be acknowledged and factored into our understanding of policy-making processes (for a related discussion, see Fforde 2019;Perl et al 2018).…”

“…In this regard, previous studies indicate that evaluations in IOs are politically contested (Højlund 2015, p. 35;Raimondo 2018, p. 32;van Voorst and Mastenbroek 2019;Weaver 2010, p. 366). If evaluation is indeed used politically, i.e.…”

“…Neglecting the political nature of evaluation stimulates unintended consequences, when functional tools are harbouring political agendas. In this regard, studies on evaluation should generally pay more attention to the setup of evaluation systems, who controls evaluation resources and how such differences came to be (see Hinterleitner et al 2016;Fforde 2019;Perl et al 2018, p. 591;van Voorst and Mastenbroek 2019).…”

“…Banner 1974;Taylor and Balloch 2005;Weiss and Jordan 1976), suggesting that evaluation results can become "ammunition in political battles" (Schoenefeld and Jordan 2019, p. 377) or that stakeholders put pressure on evaluators "to misrepresent findings" (Pleger et al 2017, p. 316). Van Voorst and Mastenbroek (2019) show that the quality of policy evaluation systematically decreases when additional political stakeholders get involved. Despite such insights, there is very little comparative evidence detailing whether policy-makers indeed realize ex-ante political interests through evaluation.…”

Delegation and stewardship in international organizations

“…Delegation literature should thus equally consider both sides of the relationship as well as the institutional context in which principals and agents interact. In addition, Paper B speaks to the public policy and evaluation literature as it highlights the importance of the political context in which public management tools such as evaluation exist Fforde, 2019;van Voorst and Mastenbroek, 2019). Neglecting the political context limits our understanding of how organizations and their policies work.…”