Can synthetic controls improve causal inference in interrupted time series evaluations of public health interventions?

Abstract

Interrupted time series designs are a valuable quasi-experimental approach for evaluating public health interventions. Interrupted time series extends a single group pre-post comparison by using multiple time points to control for underlying trends. But history bias—confounding by unexpected events occurring at the same time of the intervention—threatens the validity of this design and limits causal inference. Synthetic control methodology, a popular data-driven technique for deriving a control series from a pool of unexposed populations, is increasingly recommended. In this paper, we evaluate if and when synthetic controls can strengthen an interrupted time series design. First, we summarize the main observational study designs used in evaluative research, highlighting their respective uses, strengths, biases and design extensions for addressing these biases. Second, we outline when the use of synthetic controls can strengthen interrupted time series studies and when their combined use may be problematic. Third, we provide recommendations for using synthetic controls in interrupted time series and, using a real-world example, we illustrate the potential pitfalls of using a data-driven approach to identify a suitable control series. Finally, we emphasize the importance of theoretical approaches for informing study design and argue that synthetic control methods are not always well suited for generating a counterfactual that minimizes critical threats to interrupted time series studies. Advances in synthetic control methods bring new opportunities to conduct rigorous research in evaluating public health interventions. However, incorporating synthetic controls in interrupted time series studies may not always nullify important threats to validity nor improve causal inference.

Analysis of “Stand Your Ground” Self-defense Laws and Statewide Rates of Homicides and Firearm Homicides

“…We used negative control outcomes of suicide and firearm suicide to check for time-varying confounding. 26 We selected these outcomes because we hypothesized that they would be similarly affected by changes over time that might confound the association between SYG law enactment and changes in homicide and firearm homicide but would not be affected by the intervention itself. Such time-varying confounders included economic shifts (eg, an economic recession), changes in recording practices, and changes in firearm regulation and availability.…”

Intervention targeted at physicians’ treatment of musculoskeletal disorders and sickness certification: an interrupted time series analysis

“… 15 33 A limitation was that the intervention status was approximated (an OHS-based sick leave was preceded by a likely relevant appointment at OHS) and the information was not available for the entire follow-up period. The validity of the comparison group can be debated 17 as all employees could be prescribed sick leave both at the OHS and elsewhere (or use self-certification in short sick leaves). However, only OHS professionals were exposed to the intervention and the time series for the intervention and comparison groups were analysed separately.…”

Synthetic Control Methods for the Evaluation of Single-Unit Interventions in Epidemiology: A Tutorial

“…Even if a synthetic control closely matches the treated unit, contextual sources of bias also need to be considered when determining the appropriateness of SCM. These are the same biases that can affect most controlled before-after studies, including: 1) impacts on pre-intervention outcomes due to the anticipation of the intervention before the intervention is officially implemented ( 34 ); 2) impacts on control regions ( 35 ) (i.e., contamination/spillover effects); and 3) co-interventions or other postintervention events that do not have equivalent impact on the outcomes in the synthetic control and treated unit, confounding the effect of the intervention of interest ( 15 ). We further discuss bias related to statistical issues below.…”

“…It is therefore important to assess the risk of bias due to such events or other issues in the data. In Web Appendix 2, we discuss how placebo studies ( 12 ), manual restriction of the donor pool ( 15 ), and negative control outcomes ( 53 ) can be used to assess the robustness of the results in SCM studies.…”

“…The final section discusses ways of handling practical and methodological problems. We use data from an evaluation of Florida’s “stand your ground” law, enacted in October 1, 2005, to illustrate estimation practices and methodological considerations ( 4 , 15 ). (The law extends the right to use lethal force in self-defense to public places when threat is perceived.)…”