Inference in Differences-in-Differences with Few Treated Groups and Heteroskedasticity

Abstract

Differences-in-Differences (DID) is one of the most widely used identification strategies in applied economics. However, inference in DID models when there are few treated groups is still an open question. We show that usual inference methods used in DID models might not perform well when there are few treated groups and residuals are heteroskedastic. In particular, when there is variation in the number of observations per group, we show that inference methods designed to work when there are few treated groups would tend to (under-) over-reject the null hypothesis when the treated groups are (large) small relative to the control groups. This happens because larger groups would have lower variance, generating heteroskedasticity in the group x time aggregate DID model. We provide evidence from Monte Carlo simulations and from placebo DID regressions with the American Community Survey (ACS) dataset to show that this problem is relevant even in datasets with large number of observations per group. Then we derive alternative inference methods that provide accurate hypothesis testing in situations of few treated groups and many control groups in the presence of heteroskedasticity (including the case of only one treated group). The main assumption is that we know how the heteroskedasticity is generated, which is the case when it is generated by variation in the number of observations per group. Finally, we also show that an inference method for the Synthetic Control Estimator proposed by Abadie et al. (2010) can correct for the heteroskedasticity problem, and derive conditions under which this inference method provides accurate hypothesis testing.

Difference-in-Differences With Multiple Time Periods and an Application on the Minimum Wage and Employment

“…When there are only a few available clusters, one should look for alternative procedures, see e.g. Conley and Taber (2011) and Ferman and Pinto (2018).…”

The Local Economic Impact of a Large Research University: Evidence From Uc Merced

“…In this subsection, I examine the sensitivity of the results by running different specifications of the synthetic control method. Recently, Ferman and Pinto () have shown that asymptotic bias can exist in the synthetic control estimator when a subset of common factors is nonstationary. In this case, even a close‐to‐perfect pretreatment match for a long set of preintervention periods (which I attain when applying the synthetic control method) does not guarantee that the asymptotic bias becomes zero.…”

“…I also separately examine the employment of highly skilled subsectors in service industries, and find that the campus opening also increases the employment in highly skilled sectors. These findings are robust even when I detrend the outcome variables as suggested by Ferman and Pinto () or control for local industrial compositions.…”

“…I implement the correction technique of Ferman and Pinto () and Peri and Yasenov (), who derive conditions under which inference in synthetic control corrects for heteroscedasticity. Specifically, it requires normalizing the post‐root mean square error (RMSE) by dividing it by the pre‐RMSE.…”

An exact and robust conformal inference method for counterfactual and synthetic controls

“…2 A third major difference is that our procedure works in conjunction with many modern high-dimensional esti-mators, whereas Andrews (2003) focuses on low-dimensional GMM-type models. Hahn and Shi (2016, Section 5) informally suggest applying the end-of-sample stability test in the context of synthetic control methods and Ferman and Pinto (2017a) use a version of this test in the context of difference-in-differences approaches with few treated groups.…”