A Design-Based Perspective on Synthetic Control Methods

Abstract

Since their introduction in Abadie and Gardeazabal (2003), Synthetic Control (SC) methods have quickly become one of the leading methods for estimating causal effects in observational studies with panel data. Formal discussions often motivate SC methods by the assumption that the potential outcomes were generated by a factor model. Here we study SC methods from a design-based perspective, assuming a model for the selection of the treated unit(s), e.g., random selection as guaranteed in a randomized experiment. We show that SC methods offer benefits even in settings with randomized assignment, and that the design perspective offers new insights into SC methods for observational data. A first insight is that the standard SC estimator is not unbiased under random assignment. We propose a simple modification of the SC estimator that guarantees unbiasedness in this setting and derive its exact, randomization-based, finite sample variance. We also propose an unbiased estimator for this variance. We show in settings with real data that under random assignment this Modified Unbiased Synthetic Control (MUSC) estimator can have a root mean-squared error (RMSE) that is substantially lower than that of the difference-in-means estimator. We show that such an improvement is weakly guaranteed if the treated period is similar to the other periods, for example, if the treated period was randomly selected. The improvement is most likely to be substantial if the number of pre-treatment periods is large relative to the number of control units.

Synthetic Control as Online Linear Regression

“…As the second equality in (2) indicates, under an additional assumption that treatment timing is uniformly random, S ∼ Unif[T ], the average loss over hypothetical treatment timings is equal to the expected squared loss over S. This additional assumption is a designbased perspective (Doudchenko and Imbens (2016), Bottmer et al (2021)) on the panel causal inference problem. This perspective enables us to interpret average prediction loss over hypothetical treatment timings as expected prediction loss under the random treatment time S. The latter can in turn be thought of as design-based risk.…”

“…The previous interpretations-in ( 5) and ( 7)-rely on interpreting average loss over hypothetical values of S as expected loss over S, which requires uniform treatment timing S ∼ Unif[T ]. Despite being plausible in certain settings and appearing elsewhere in the literature (Doudchenko and Imbens (2016), Bottmer et al (2021)), this assumption is perhaps crude. 21 To some extent, this is inevitable: Since we are agnostic on the outcome generation process, it is unavoidable to make treatment timing assumptions in order to obtain nontrivial statistical results on estimation of causal quantities.…”

Synthetic Controls for Experimental Design

“…Agarwal et al (2021) propose synthetic interventions, a framework related to synthetic controls, and apply it to estimate treatment effect heterogeneity in experimental setting with multiple treatments. Bottmer et al (2021) is also related to the present article in the sense that they study synthetic control estimation in an experimental setting. Their article, however, considers only the case when the treatment is randomized, and is not concerned with issues of experimental design.…”

Synthetic Control As Online Linear Regression

“…The previous interpretations rely heavily on interpreting average loss over time as expected loss over τ (( 5) and ( 7)), which requires uniform treatment timing τ ∼ Unif[T ]. Despite being plausible in certain settings and appearing elsewhere in the literature (Doudchenko and Imbens, 2016;Bottmer et al, 2021), this assumption is perhaps not entirely palatable. 15 Note, however, such an assumption is only necessary to interpret average losses as expected losses, and the a priori position that it is reasonable to expect a causal estimator to perform well relative to a set of oracles, at least on average over time, strikes us as defensible.…”

“…1 See the review by Abadie (2021) as well as the special section on synthetic control methods in the Journal of the American Statistical Association. (Abadie and Cattaneo, 2021) 2 Notably, similar to this paper, Bottmer et al (2021) consider a design-based framework which conditions on the outcomes and consider randomness arising solely from assignment of the treated unit and the treatment time period.…”