Cherry Picking with Synthetic Controls

Abstract

We evaluate whether a lack of guidance on how to choose the matching variables used in the Synthetic Control (SC) estimator creates specification-searching opportunities. We provide theoretical results showing that specification-searching opportunities are asymptotically irrelevant if we restrict to a subset of SC specifications. However, based on Monte Carlo simulations and simulations with real datasets, we show significant room for specification searching when the number of pre-treatment periods is in line with common SC applications, and when alternative specifications commonly used in SC applications are also considered. This suggests that such lack of guidance generates a substantial level of discretion in the choice of the comparison units in SC applications, undermining one of the advantages of the method. We provide recommendations to limit the possibilities for specification searching in the SC method. Finally, we analyze the possibilities for specification searching and provide our recommendations in a series of empirical applications. C 2020 by the Association for Public Policy Analysis and Management on behalf of the Association for Public Policy Analysis and Management variables, the choice of how to split the pre-treatment periods into training and validation periods, and even the choice of software and data-sorting criteria (see Klößner et al., 2017, for details on this last point). Therefore, our results should be seen as a lower bound on the possibilities for specification searching in SC applications. We focus on the choice of pre-treatment outcome lags, rather than on the inclusion of covariates, because it is possible to systematically analyze the inclusion of pre-intervention outcomes lags in a way that encompasses all applications, while covariates may differ in complex ways from one application to another. We consider the possibility of specification searching in the decision to include covariates in our empirical applications. 6 Appendices are available at the end of this article as it appears in JPAM online. Go to the publisher's website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 7 In Appendix B, we also consider specifications that use time-invariant covariates as predictors, in addition to functions of the pre-treatment outcomes. All results remain similar. This package solves the nested minimization problem described by equations (1) and (2). We specify the optimization method to be BFGS only and use optimization routine Low Rank Quadratic Programming when Interior Point optimization routine does not converge. 24 When we adopt this decision rule in our MC simulations, then the probability of rejecting the null at 5 percent for all specifications is lower than 1 percent in all scenarios. If we discard specifications 6 and 7, then this rejection rate ranges from 1 percent when T 0 = 12 to 2.8 percent when T 0 = 400. 25 Following the best practices in terms of transparency and replicability, Hainmueller (2014) made their dataset and replication ...

Transparency, Reproducibility, and the Credibility of Economics Research

“…Several psychology journals currently have begun using results-blind review, either regularly or in special issues (Chambers 2013;Chambers et al 2014;Nosek and Lakens 2014). 31 A recent issue of Comparative Political Studies was the first to our knowledge to feature results-blind review in political science (Findley et al 2016).…”

“…This issue has been extensively discussed in recent years within medical research, but consensus has not yet been reached in that community. It actually appears that some of the most prestigious medical research journals, which typically publish randomized trials, are even more in favor of the registration of observational studies than the editors of journals that publish primarily in nonexperimental research (see the dueling editorial statements in Epidemiology 31 A list of journals that have adopted registered reports is available at: https://osf.io/8mpji/wiki/home/.…”

Assessing the Causal Effect of Binary Interventions from Observational Panel Data with Few Treated Units

“…When implementing the SCM, users need to decide which variables (preintervention outcomes, covariates or summaries of these) to use to determine the weights. Ferman, Pinto and Possebom (2016) demonstrate that the estimated counterfactual may differ depending on which variables are used. Dube and Zipperer (2015) develop the following approach for selecting among K sets of variables.…”

Social democratic representation and welfare spending: a quantitative case study

“…This is a useful restriction because the linear combination of the control units to match Fl a might involve a lot of interpolation, and hence bias, if donor pool units are very dierent (Abadie et al 2010: 495). Since the administrative error is unlikely to have happened in a large municipality, not oer much guidance on this issue, and the lack of guidance can result in considerable researcher degrees of freedom to pick the specication that yields the preferred conclusions (Ferman, Pinto, and Possebom 2017). To reduce this worry I estimate a set of dierent specications and use decision rules to pick specications for further analyses (see Ferman, Pinto, and Possebom 2017; Dube and Zipperer 2015).…”