Moving beyond the classic difference-in-differences model: a simulation study comparing statistical methods for estimating effectiveness of state-level policies

Abstract

Background Reliable evaluations of state-level policies are essential for identifying effective policies and informing policymakers’ decisions. State-level policy evaluations commonly use a difference-in-differences (DID) study design; yet within this framework, statistical model specification varies notably across studies. More guidance is needed about which set of statistical models perform best when estimating how state-level policies affect outcomes. Methods Motivated by applied state-level opioid policy evaluations, we implemented an extensive simulation study to compare the statistical performance of multiple variations of the two-way fixed effect models traditionally used for DID under a range of simulation conditions. We also explored the performance of autoregressive (AR) and GEE models. We simulated policy effects on annual state-level opioid mortality rates and assessed statistical performance using various metrics, including directional bias, magnitude bias, and root mean squared error. We also reported Type I error rates and the rate of correctly rejecting the null hypothesis (e.g., power), given the prevalence of frequentist null hypothesis significance testing in the applied literature. Results Most linear models resulted in minimal bias. However, non-linear models and population-weighted versions of classic linear two-way fixed effect and linear GEE models yielded considerable bias (60 to 160%). Further, root mean square error was minimized by linear AR models when we examined crude mortality rates and by negative binomial models when we examined raw death counts. In the context of frequentist hypothesis testing, many models yielded high Type I error rates and very low rates of correctly rejecting the null hypothesis (< 10%), raising concerns of spurious conclusions about policy effectiveness in the opioid literature. When considering performance across models, the linear AR models were optimal in terms of directional bias, root mean squared error, Type I error, and correct rejection rates. Conclusions The findings highlight notable limitations of commonly used statistical models for DID designs, which are widely used in opioid policy studies and in state policy evaluations more broadly. In contrast, the optimal model we identified--the AR model--is rarely used in state policy evaluation. We urge applied researchers to move beyond the classic DID paradigm and adopt use of AR models.

The impact of Illinois’ comprehensive handheld phone ban on talking on handheld and handsfree cellphones while driving

“…35 We therefore did not need to account for temporal correlation in the data, which can improve DID inference for longitudinal data. 32,36…”

“…This approach facilitated greater ease in interpreting the interaction term on the probability scale. 29 We are aware of several recent methodological developments in policy evalutaion, [30][31][32][33] many of which build upon the seminal synthetic control method of Abadie et al (2010). 34 These methods could have been employed in this setting to create a more suitable control group and alleviate potential bias that has been observed for traditional DID in certain settings.…”

Identifying Optimal Methods for Addressing Confounding Bias When Estimating the Effects of State-level Policies

“…The autoregressive model we examined-highly rated in prior simulation studies, 6,13 included a single lagged value of the outcome expressed as:…”

“…Numerous studies have highlighted the challenges of using the common two-way fixed effects DID model to estimate policy effects in the presence of treatment effect heterogeneity. Recent methodologic advances have sought to improve the estimation of policy effects in the presence of staggered adoption, including the use of autoregressive models 13 ; augmented synthetic control methods 14 ; and methods from Callaway–Sant’Anna, 15 which directly allow for effect heterogeneity over time and by state. Augmented synthetic control and Callaway–Sant’Anna methods take a more design-based approach (rather than regression-based) to policy evaluation, which helps mitigate concerns about bias that stems from controlling for time-varying post-treatment confounding variables directly in the model.…”

Correction to: Moving beyond the classic diferencein-diferences model: a simulation study comparing statistical methods for estimating efectiveness of state-level policies

“…Following publication of the original article [ 1 ], the authors noticed a typographical error on the name of one of the authors on this article. The author name “Bradley D. Stei should be Bradley D. Stein”.…”