Distributional Analysis in Educational Evaluation: A Case Study from the New York City Voucher Program

Abstract

We use quantile treatment effects estimation to examine the consequences of the random-assignment New York City School Choice Scholarship Program (NYCSCSP) across the distribution of student achievement. Our analyses suggest that the program had negligible and statistically insignificant effects across the skill distribution. In addition to contributing to the literature on school choice, the paper illustrates several ways in which distributional effects estimation can enrich educational research: First, we demonstrate that moving beyond a focus on mean effects estimation makes it possible to generate and test new hypotheses about the heterogeneity of educational treatment effects that speak to the justification for many interventions. Second, we demonstrate that distributional effects can uncover issues even with well-studied datasets by forcing analysts to view their data in new ways. Finally, such estimates highlight where in the overall national achievement distribution test scores of children exposed to particular interventions lie; this is important for exploring the external validity of the intervention’s effects.

What differences a day can make: Quantile regression estimates of the distribution of daily learning gains

“…Specifically, the result that additional instructional time are more beneficial to higher achievers has implications for the growth and persistence of achievement gaps (Bitler et al, 2015) and is consistent with the theory that "skills beget skills." Identifying the mechanisms underlying these results is outside the scope of the current study, but would be useful for future research to consider.…”

“…While average effects are interesting and add to our understanding of the relationship between instructional time and student outcomes, they overlook potential variation across the achievement distribution in the relationship between instructional time and student achievement (Bitler, Domina, Penner, and Hoynes, 2015;Eide and Showalter, 1998). We begin to fill this gap in the literature by extending the identification strategy pioneered by Fitzpatrick et al (2011) to the quantile regression context.…”

Teaching for All? Teach For America's Effects Across the Distribution of Student Achievement

“…The inverse-propensity score weighted quantile treatment effects, which are described in more detail above, produce unconditional estimates of the impact of TFA on the distribution of student achievement, adjusting for the block randomization design, baseline control variables, previously identified missing values, and the previously unidentified missing values described above (c.f. Bitler, Domina, Penner, & Hoynes, 2015; Bitler, Gelbach, & Hoynes, 2006; Firpo et al, 2009). The process of conditioning, or including covariates, “shifts” an observation’s placement in the conditional distribution (Powell, 2010).…”

Does Regulation Induce Homogenization? An Analysis of Three Voucher Programs in the United States

“…To date, researchers have conducted 16 evaluations of the impact of school choice on participants' academic achievement using random assignment methods in the United States. Of those 16 evaluations, 10 found statistically significant positive gains on student achievement for some or all students (Rouse, 1998;Greene, Peterson, & Du, 1999;Greene, 2000;Howell, Wolf, Campbell, & Peterson, 2002 (contains three studies); Barnard et al, 2003;Cowen, 2008;Jin, Barnard, & Rubin, 2010;Bitler et al, 2015), four found no impact (Krueger & Zhu, 2004;Bettinger & Slonim, 2006;Wolf et al, 2013;, and two found negative impacts on academic achievement as a result of voucher use (Abdulkadiroglu, Pathak, & Walters, 2015;Dynarski et al, 2017).…”