Experimental evidence, scaling and public policy: a perspective from developing countries

Abstract

I highlight two important factors particular to less-developed countries that can bias evidence generation and contribute to the ‘voltage drop’ in programme benefits, moving from field research experiments to policy implementation at scale. The first is the non-linear increase in information processing and coordination costs associated with upscaling in less-developed countries, given limited state capacity and rigid organizational hierarchies. The second is political bias in the choice of programmes considered for rigorous evaluation itself, resulting in distorted evidence and policy choice. These two factors raise considerations that complement the economics-based approach outlined by Al-Ubaydli et al. in the quest for more rigorous, evidence-based policy.

A rejoinder: ‘How can experiments play a greater role in public policy? Twelve proposals from an economic model of scaling’

“…We also agree on the importance of the politics in the project selection process, and many of the issues raised by Mani (2020) reflect those raised by Ho et al (2020), John (2020) and Supplee and Kane (2020), so we do not add further comments here.…”

“…Ho et al 's (2020) Suggestion #5 aligns perfectly with our checklist (e.g., see the section for researchers within the checklist) that pushes for greater transparency, particularly regarding the information that concerns the characteristics of the population and the situation of the experimental study. Finally, Suggestion #6 echoes in part comments made by John (2020), Mani (2020) and Supplee and Kane (2020), so we will not expand upon this here other than to point the reader to our thoughts laid out above in the introduction to this rejoinder.…”

“…Leveraging insights from the political science literature, John (2020) delves deeper into our footnote 9, noting, “Bureaucrats too are political animals whose interests are tied to those of the politicians who usually command them and who have an interest in the preservation of their bureaus and in promoting their own careers.” We acknowledge that capturing this political dynamic was beyond the scope of our work. Many of the excellent issues raised were pointed out by Ho et al (2020), Mani (2020) and Supplee and Kane (2020).…”

“…Mani (2020) provides a wonderful exposition in applying the scale-up effect framework from the perspective of developing countries and she points out two key issues to be considered: the “limited state capacity” and the “politics of project selection.”…”

“…At this point, the astute reader might stop and pause to wonder why so many of the pundits focused on the third link in our statement (see Ho et al , 2020; John, 2020; Mani, 2020; Supplee & Kane, 2020). In this context, the focus for some reviewers seemed to be on what we did not model rather than what we did.…”