The Sequential Scale-Up of an Evidence-Based Intervention: A Case Study

Abstract

Policy makers face dilemmas when choosing a policy, program, or practice to implement. Researchers in education, public health, and other fields have proposed a sequential approach to identifying interventions worthy of broader adoption, involving pilot, efficacy, effectiveness, and scale-up studies. In this article, we examine a scale-up of an early math intervention to the state level, using a cluster randomized controlled trial. The intervention, Pre-K Mathematics, has produced robust positive effects on children's math ability in prior pilot, efficacy, and effectiveness studies. In the current study, we ask if it remains effective at a larger scale in a heterogeneous collection of pre-K programs that plausibly represent all low-income families with a child of pre-K age who live in California. We find that Pre-K Mathematics remains effective at the state level, with positive and statistically significant effects (effect size on the Early Childhood Longitudinal Study, Birth Cohort Mathematics Assessment = .30, p < .01). In addition, we develop a framework of the dimensions of scale-up to explain why effect sizes might decrease as scale increases. Using this framework, we compare the causal estimates from the present study to those from earlier, smaller studies. Consistent with our framework, we find that effect sizes have decreased over time. We conclude with a discussion of the implications of our study for how we think about the external validity of causal relationships.

Red Light, Purple Light! Results of an Intervention to Promote School Readiness for Children From Low-Income Backgrounds

“…Examples of interventions include the preschool Promoting Alternative Thinking Strategies (PATHS) curriculum, which focuses on children’s problem solving skills, emotional awareness, social-emotional skills and self-control (Greenberg and Kusche, 1993; Kam et al, 2003; Domitrovich et al, 2007). There are also interventions that focus explicitly on improving preschoolers’ early math that have been shown to be effective such as Pre-K Mathematics (Starkey et al, 2004; Thomas et al, 2018) and Building Blocks (Clements and Sarama, 2007; Clements and Sarama, 2011). Finally, interventions designed to promote preschoolers’ emergent literacy have shown positive effects (Justice and Pullen, 2003; Farver et al, 2009; Justice et al, 2009).…”

A multi-sensory tutoring program for students at risk of reading difficulties: Evidence from a randomized field experiment

“…We provided minimal extra support and monitoring during the implementation, which mimicked how the program is implemented when schools use it outside the study. We expect this feature to result in smaller effect sizes (see e.g., Thomas et al, 2018, for an interesting discussion and results in line with this hypothesis). All comparison studies monitor and support their tutors more than we do or have personnel hired by the study that perform the intervention.…”

How much bias results if a quasi-experimental design combines local comparison groups, a pretest outcome measure and other covariates?: A within study comparison of preschool effects.

“…The intervention we examine is Pre-K Mathematics, a supplementary math curriculum for use with economically disadvantaged preschool children. The program, recently described in Thomas, Cook, Klein, Starkey, and DeFlorio (2018), combines a developmentally sequenced curriculum of small-group math activities with concrete manipulatives that teachers implement in the classroom in order to improve skills in numbers, operations, geometry, pattern knowledge, and informal measurement. To improve fidelity and dosage, teachers receive professional development first through workshops and then through in-class coaching.…”

“…The ultimate goal is to prepare children for standards-based math instruction in kindergarten. The What Works Clearinghouse (2013) rates Pre-K Mathematics as an effective evidence-based program, and several more recent RCTs have shown that its improvements in math performance are of at least moderate effect size (Thomas et al, 2018).…”

“…The data used in this WSC come from the cluster RCT described in Thomas et al (2018) in which pre-K classrooms were randomly assigned to treatment or control status throughout California. Children in the treatment classrooms received Pre-K Mathematics , while those in the control classrooms received business-as-usual math instruction.…”