The Perils of Counterfactual Analysis with Integrated Processes

Abstract

Recently, there has been a growing interest in developing econometric tools to conduct counterfactual analysis with aggregate data when a "treated" unit suffers an intervention, such as a policy change, and there is no obvious control group. Usually, the proposed methods are based on the construction of an artificial counterfactual from a pool of "untreated" peers, organized in a panel data structure. In this paper, we investigate the consequences of applying such methodologies when the data are formed by integrated process of order 1. We find that without a cointegration relation (spurious case) the intervention estimator diverges resulting in the rejection of the hypothesis of no intervention effect regardless of its existence. Whereas, for the case when at least one cointegration relation exists, we have a √ T-consistent estimator for the intervention effect albeit with a non-standard distribution. However, even in this case, the test of no intervention effect is extremely oversized if nonstationarity is ignored. When a drift is present in the data generating processes, the estimator for both cases (cointegrated and spurious) either diverges or is not well defined asymptotically. As a final recommendation we suggest to work in first-differences to avoid spurious results.

Synthetic Controls with Imperfect Pre-Treatment Fit

“…We show in Appendix A.5.3 that our main result that the original and the demeaned SC estimators are generally asymptotically biased if there are unobserved time-varying confounders (Propositions 1 and 2) still applies if we also relax the non-negative and the adding-up constraints, which essentially leads to the panel data approach suggested by Hsiao et al (2012), and further explored by Li and Bell (2017). 19 Our conditions for unbiasedness of the SC estimator also apply to the estimators proposed by Carvalho et al (2018) and Carvalho et al (2016) when J is fixed.…”

“…Therefore, the asymptotic bias we find for the SC estimator is consistent with the results from a large literature on factor models. We also revisit the conditions for validity of alternative estimators, such as the ones proposed by Hsiao et al (2012), Li and Bell (2017), Carvalho et al (2018) and Carvalho et al (2016). We show that these papers rely on assumptions that implicitly imply no selection on unobservables, which clarifies why their consistency/unbiasedness results when J is fixed do not contradict the literature on factor models.…”

“…The asymptotic bias we derive for the SC estimator does not rely on the fact that the SC unit is constrained to convex combinations of control units, so it also applies to other related panel data approaches that have been studied in the context of an imperfect pre-treatment fit, such as Hsiao et al (2012), Li and Bell (2017), Carvalho et al (2018), Carvalho et al (2016), . We show that these papers rely on assumptions that essentially exclude the possibility of selection on unobservables.…”

ArCo: An R package to Estimate Artificial Counterfactuals

“…On the other hand, the trend-stationarity assumption has been dangerously ignored in the literature. Carvalho et al (2016a) investigate the consequences of applying the ArCo, SC or PF methods when the data are integrated processes of order 1. They find that without a cointegrating relationship (spurious case) the estimator of the effects of the intervention diverges, resulting in the rejection of the null hypothesis of no effect regardless of its existence with probability approaching 1.…”

“…This section aims to describe the mathematical notation and the key definitions of the ArCo methodology in a way that is compatible with the ArCo (Fonseca et al, 2017) package. For further details on statistical properties and theoretical results, see Carvalho et al (2016b) and Carvalho et al (2016a). Everything concerning the technique used to estimate the first-step model is left in a very general way as the ArCo package was developed to accept many different classes of models.…”