External Validity in Policy Evaluations That Choose Sites Purposively

Abstract

Evaluations of the impact of social programs are often carried out in multiple “sites,” such as school districts, housing authorities, local TANF offices, or One-Stop Career Centers. Most evaluations select sites purposively following a process that is nonrandom. Unfortunately, purposive site selection can produce a sample of sites that is not representative of the population of interest for the program. In this paper, we propose a conceptual model of purposive site selection. We begin with the proposition that a purposive sample of sites can usefully be conceptualized as a random sample of sites from some well-defined population, for which the sampling probabilities are unknown and vary across sites. This proposition allows us to derive a formal, yet intuitive, mathematical expression for the bias in the pooled impact estimate when sites are selected purposively. This formula helps us to better understand the consequences of selecting sites purposively, and the factors that contribute to the bias. Additional research is needed to obtain evidence on how large the bias tends to be in actual studies that select sites purposively, and to develop methods to increase the external validity of these studies.

Characteristics of School Districts That Participate in Rigorous National Educational Evaluations

“…IES also sponsors large-scale evaluations through contracts to major research firms (e.g., Abt Associates, Mathematica Policy Research, MDRC). These evaluations usually select a sample designed to cover all regions of the country, but sites are selected purposively to reduce costs and sometimes with other objectives in mind (e.g., to test the intervention in sites where it will produce the greatest “contrast” between the treatment and control conditions, suggesting that it may have the greatest impact); they are rarely selected randomly to be formally representative of any broader population of potential interest to policymakers (Olsen et al, 2013). …”

Social Experiments

“…In fact, experiments are frequently conducted in only one or two localities that cannot claim to be representative of the nation, state, or other jurisdiction for which policy is made. (See Olsen et al (2013) for a derivation of the bias that may occur when experiments are conducted with nonrepresentative populations. )…”

Sample Selection in Randomized Experiments: A New Method Using Propensity Score Stratified Sampling

“…The average treatment effect is typically estimated using the n units in the sample and a multilevel model that accounts for the study design (e.g., random block design or cluster randomized design; Raudenbush & Bryk, 2002), and the impact of the experiment is evaluated by comparing this estimate to its standard error. Imai, King, and Stuart (2008) and Olsen et al (2012) have shown that when the sample is not representative of the population and when site-specific treatment effects vary, the sample based estimate of is biased. Our goal here, therefore, is to develop a strategy for selecting the n units in the sample S so that the sample is compositionally similar to the N units in the inference population P, thereby leading to a less biased and more precise estimate of the population average treatment effect.…”