What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results

Abstract

Policymakers often consider interventions at the scale of the population, or some other large scale. One of the sources of information about the potential effects of such interventions is experimental studies conducted at a significantly smaller scale. A common occurrence is for the treatment effects detected in these small-scale studies to diminish substantially in size when applied at the larger scale that is of interest to policymakers. This paper provides an overview of the main reasons for a breakdown in scalability. Understanding the principal mechanisms represents a first step toward formulating countermeasures that promote scalability.

The Dozen Things Experimental Economists Should Do (More of)

“…Second, whereas that literature tends to focus on program fidelity as a major reason for the lack of proper scaling, we see three main areas where challenges to scalability arise: statistical inference, representativeness of the population and representativeness of the situation. Al‐Ubaydli, List, and Suskind () provide a formal model of the way these three factors manifest in the market for scientific knowledge; we simply sketch them below to highlight issues experimenters should consider in their design, analysis, and interpretation that could affect the scalability of their results.…”

“…During scaling, the population that selects into the program is often different than the original experimental sample, raising the concern that the estimated treatment effect will be different (usually smaller) in the new population. Such “scaling bias” may result from adverse heterogeneity (Al‐Ubaydli, List, and Suskind ), describing a situation when the original experimental participants' attributes are correlated with higher expected outcomes. This may occur as a result of participation bias (participants self‐select into the experiment on the basis of their expected gains from participation Al‐Ubaydli and List ) or publication bias (researchers have incentives to find participants who yield large treatment effects Al‐Ubaydli, List, and Suskind ).…”

“…Such “scaling bias” may result from adverse heterogeneity (Al‐Ubaydli, List, and Suskind ), describing a situation when the original experimental participants' attributes are correlated with higher expected outcomes. This may occur as a result of participation bias (participants self‐select into the experiment on the basis of their expected gains from participation Al‐Ubaydli and List ) or publication bias (researchers have incentives to find participants who yield large treatment effects Al‐Ubaydli, List, and Suskind ). Another concern for scalability related to the population is attrition, an issue we discussed in detail in section 3.1.…”

“…There are many possible reasons why such a voltage drop might occur, and it is crucial to understand why scaled-up programs often do not work as intended. Yet we believe that the problem is currently too narrowly defined along two important dimensions (Al-Ubaydli, List, and Suskind 2017c). First, the discussion around voltage effects in the implementation science literature tends to focus mostly on the scaled-up program's benefits.…”

“…Interestingly, many of these issues could be exacerbated “when rolling out revolutionary ideas, as these often challenge the power and established practices of incumbent organizations” (Al‐Ubaydli, List, and Suskind ). Therefore, programs with greater community coalition functions, communication to stakeholders, and sustainability are more likely to still be in place over two or more years beyond their original funding (Cooper, Bumbarger, and Moore ).…”