From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application

Abstract

The promise of randomized controlled trials is that evidence gathered through the evaluation of a specific program helps us—possibly after several rounds of fine-tuning and multiple replications in different contexts—to inform policy. However, critics have pointed out that a potential constraint in this agenda is that results from small “proof-of-concept” studies run by nongovernment organizations may not apply to policies that can be implemented by governments on a large scale. After discussing the potential issues, this paper describes the journey from the original concept to the design and evaluation of scalable policy. We do so by evaluating a series of strategies that aim to integrate the nongovernment organization Pratham’s “Teaching at the Right Level” methodology into elementary schools in India. The methodology consists of reorganizing instruction based on children’s actual learning levels, rather than on a prescribed syllabus, and has previously been shown to be very effective when properly implemented. We present evidence from randomized controlled trials involving some designs that failed to produce impacts within the regular schooling system but still helped shape subsequent versions of the program. As a result of this process, two versions of the programs were developed that successfully raised children’s learning levels using scalable models in government schools. We use this example to draw general lessons about using randomized control trials to design scalable policies.

The Dozen Things Experimental Economists Should Do (More of)

“…Moreover, risk averse individuals might be less likely to enroll in an experiment (Al‐Ubaydli and List ; Heckman ). Participation bias may also arise in the field, because organizations who agree to collaborate with researchers in an experiment are usually exceptional (Banerjee et al ). Consider the example of Behaghel, Crépon, and Barbanchon (), where French firms could opt into an experiment that randomly anonymized the resumes they received from job applicants.…”

“…Moreover, program evaluations are often run by particularly high‐quality officials or NGOs in a way that is hardly possible to scale up: as the program is expanded, its quality may deteriorate (Duflo, Glennerster, and Kremer ). When scaling, researchers need to be aware that more workers must be hired, and they might be of lower quality or have less interest in the program, simply due to the diseconomies of scale associated with requiring more labor in a labor market with inelastically supplied human capital (Al‐Ubaydli, List, and Suskind ; Davis et al ; Banerjee et al ). This may reduce the benefits of the program (through lower‐quality program workers) and/or increase its costs (when trying to keep the quality of workers high).…”

“…Finally, it is important to evaluate the program's implementation, and document all steps of the process (Duflo, Glennerster, and Kremer ; Banerjee et al ). This includes having a preanalysis plan, and creating an initial program that is modular, in the sense that its implementation can described by a simple protocol (Banerjee ).…”

“…A prominent example is that of Miguel and Kremer () who realized the importance of spillover effects from deworming programs by randomizing the programs at the school (rather than the student) level. However, Banerjee et al () warn that sometimes it is difficult to know ex ante what randomization unit will be large enough to capture all GE effects.…”

Toward High‐Quality Early Childhood Development Programs and Policies at National Scale: Directions for Research in Global Contexts

“…However, several studies in the global evidence base have shown that when smaller-scale programs go to a larger scale, positive effects are often diminished or lost (for example, Bold, Kimenyi, Mwabu, Ng'ang'a, & Sandefur, 2013;Dodge, 2011). On the other hand, some recent examples show success when the process of scaling involves multiple stages and testing in a variety of local policy and implementation contexts-that is, when across iterations the core of a program's active ingredients are maintained in adaptations to larger systems-level factors, such as workforce capacity, local policy, supervision and governance (Banerjee et al, 2016). "Small to bigger" expansion may be more successful if planning for scale occurs starting during the phase of small-scale implementation (Dodge, 2011;Supplee & Metz, 2015).…”