Inference with Difference-in-Differences Revisited

Abstract

Abstract:A growing literature on inference in difference-in-differences (DiD) designs has been pessimistic about obtaining hypothesis tests of the correct size, particularly with few groups. We provide Monte Carlo evidence for four points: (i) it is possible to obtain tests of the correct size even with few groups, and in many settings very straightforward methods will achieve this; (ii) the main problem in DiD designs with grouped errors is instead low power to detect real effects; (iii) feasible GLS estimation combined with robust inference can increase power considerably whilst maintaining correct test size -again, even with few groups, and (iv) using OLS with robust inference can lead to a perverse relationship between power and panel length.

Scars of recessions in a rigid labor market

“…If, as in our case, the number of clusters is small (40 in the provincial model and 8 in the national model), 19 it is very difficult to obtain correctly sized tests (Angrist and Pischke, 2009;Cameron et al, 2008). In Section 5.2 we propose a new inference procedure that builds on the work of Brewer et al (2013) and Wooldridge (2006Wooldridge ( , 2010, but that is modified to take the specificities of our data into account.…”

“…25 Moreover, the method does not exploit the possibility of enhancing the power of the statistical tests. Brewer et al (2013) recently proposed a straightforward method for inference that addresses these limitations in a differencein-differences (DiD) design. They demonstrate in Monte Carlo analysis that correctly sized tests can be obtained by using bias corrected clustered standard errors in an ordinary least squares (OLS) regression of the covariate-adjusted group-time means of the dependent variable on the covariates varying at the group-time level.…”

“…each combination of graduation year (g) and province (p) is a cluster, at the time level (t = g + e), and at the group-time level (gpt), this approach can be applied to our analysis. However, in contrast to Brewer et al (2013), group-time cells in our sample contain a relatively small number of observations, so that we cannot ignore measurement error in the covariate-adjusted group-time means of the dependent variables. To generalize their approach, we build on the work of Wooldridge (2006Wooldridge ( , 2010.…”

“…Wooldridge proposes a FGLS estimator in case of cross-sectional data with only measurement error and no unobserved group effects. We adjust this method for panel data and show how, as in Brewer et al (2013), autocorrelated unobserved group effects can be integrated in this approach.…”

“…Bertrand et al (2004) demonstrate that in such circumstances the standard error can be severely downwards biased and that this bias can be very severe if the number of groups (clusters) is small. To address this problem, we propose a novel feasible generalized least squares (FGLS) method closely related to the efficient Minimum Distance (or Minimum Chi-square) estimator of Wooldridge (2006Wooldridge ( , 2010 for cross-section data and the FGLS recently introduced by Brewer et al (2013) for difference-in-differences designs.…”