Using the Results from Rigorous Multisite Evaluations to Inform Local Policy Decisions

Abstract

Evidence‐based policy at the local level requires predicting the impact of an intervention to inform whether it should be adopted. Increasingly, local policymakers have access to published research evaluating the effectiveness of policy interventions from national research clearinghouses that review and disseminate evidence from program evaluations. Through these evaluations, local policymakers have a wealth of evidence describing what works, but not necessarily where. Multisite evaluations may produce unbiased estimates of the average impact of an intervention in the study sample and still produce inaccurate predictions of the impact for localities outside the sample for two reasons: (1) the impact of the intervention may vary across localities, and (2) the evaluation estimate is subject to sampling error. Unfortunately, there is relatively little evidence on how much the impacts of policy interventions vary from one locality to another and almost no evidence on the implications of this variation for the accuracy with which the local impact of adopting an intervention can be predicted using findings from an evaluation in other localities. In this paper, we present a set of methods for quantifying the accuracy of the local predictions that can be obtained using the results of multisite randomized trials and for assessing the likelihood that prediction errors will lead to errors in local policy decisions. We demonstrate these methods using three evaluations of educational interventions, providing the first empirical evidence of the ability to use multisite evaluations to predict impacts in individual localities—i.e., the ability of “evidence‐based policy” to improve local policy.

A Within-Study Approach to Evaluating the Role of Moderators of Impact in Limiting Generalizations from “Large to Small”

“…Second, the reductions in impact heterogeneity from modeling moderators are not statistically significant based on changes in the deviance statistic, and they range in magnitude from small to moderate (0.02 [5%] to 0.08 [32%] SDs, depending on the model). The results observed here are similar in magnitude to those obtained by Orr et al (2019).…”

“…studies can be reconceptualized as a form of random "non-experimental mismatch error" (Bloom et al, 2005). A different but related approach that addresses a similar question is presented in Orr et al (2019).…”

From Local to Global: External Validity in a Fertility Natural Experiment

“…Our analysis addresses both micro-level differences and macro-level differences (that is, country-yearlevel contextual characteristics). In a recent study that comes closer to what we do here, Orr et al (2017) work with the results of multisite education experiments, using a leaveone-out approach to examine the out-of-sample predictive accuracy of multilevel mixedeffects regression models that model treatment effects as functions of context-level variables. In a similar spirit, Bloom et al (2016) use a multilevel mixed-effects regression model to estimate the variance of treatment effects across sites.…”

“…Recall that in expression (5) we defined G H as the estimator for the target population treatment effect and in expression (6) We estimate the total variance (U V W + U H W ) in a manner that accounts for potential dependency between this variance and covariates. We do so using a leave-one-out approach, similar to that of Orr et al (2017). We first generate predictions, G H( , for each of the reference contexts in the evidence base, and then we take the difference R ( = G H( − G ( , where G ( is the effect estimated using the natural experiment in context ".…”

Balancing Rigor with Complexity in Understanding the Impacts of Child Maltreatment Prevention Programs

“…Despite the recognized value and purpose of randomized trials and the oft-stated preference of funders and researchers for large-scale, multi-site controlled trials, there is increasing recognition of the challenges of translating the results of these types of studies to site-level work. This is partly not because of the pragmatic issues described here but also because averaging across sites can mask the variation seen between sites, reducing applicability of a global finding to the local setting (Orr et al 2019). One result is that actual interventions that "work" in community contexts remain few and far between, and we cannot solve the problems of child maltreatment by being complacent with short lists of evidence-based prevention practices that have not been sufficiently field-tested.…”