Examination of the Synthetic Control Method for Evaluating Health Policies with Multiple Treated Units

Abstract

This paper examines the synthetic control method in contrast to commonly used difference‐in‐differences (DiD) estimation, in the context of a re‐evaluation of a pay‐for‐performance (P4P) initiative, the Advancing Quality scheme. The synthetic control method aims to estimate treatment effects by constructing a weighted combination of control units, which represents what the treated group would have experienced in the absence of receiving the treatment. While DiD estimation assumes that the effects of unobserved confounders are constant over time, the synthetic control method allows for these effects to change over time, by re‐weighting the control group so that it has similar pre‐intervention characteristics to the treated group.We extend the synthetic control approach to a setting of evaluation of a health policy where there are multiple treated units. We re‐analyse a recent study evaluating the effects of a hospital P4P scheme on risk‐adjusted hospital mortality. In contrast to the original DiD analysis, the synthetic control method reports that, for the incentivised conditions, the P4P scheme did not significantly reduce mortality and that there is a statistically significant increase in mortality for non‐incentivised conditions. This result was robust to alternative specifications of the synthetic control method. © 2015 The Authors. Health Economics published by John Wiley & Sons Ltd.

Measuring the Vice-Presidential Home State Advantage With Synthetic Controls

“…In addition, Dudley and Rapoport and Devine and Kopko erroneously include "treated" elections in the five pre-treatment elections. Kreif et al, 2015;Liou & Musgrave, 2014). For a unifying approach to the synthetic control method, see Xu (2015).…”

Do mergers of large local governments reduce expenditures? – Evidence from Germany using the synthetic control method

“…19 In these cases, I do not find a significant effect of mergers on expenditures. If pre-reform trends vary, the synthetic control group method is more reliable (Kreif et al 2016). In these cases, the difference between synthetic control group method predicted expenditures and realized expenditures, i.e., the reform effect, is fairly small for administrative expenditures (2.32 Euro per capita) and education expenditures (-1.56 Euro per capita).…”

“…The synthetic control method is a powerful tool to evaluate policy reforms if the number of treated units is small, and only aggregated outcomes are observable (see also Abadie et al 2015). I show that the method is superior to difference-in-differences estimations, when the common trend assumption seems to be violated (see also Kreif et al 2016).…”

Impact of Provider Incentives on Quality and Value of Health Care

“…When HQID was transferred to England, there was an apparent short-term reduction in mortality for patients admitted with one of the three targeted conditions (pneumonia) (94), but this result was not sustained after the second year (63). Furthermore, reanalysis of the data suggested not only that the initial reduction was not statistically significant but also that mortality rates for nonincentivized conditions increased (62). Similarly, mortality rates for conditions not incentivized under the QOF increased in the United Kingdom relative to other countries during the early years of the scheme, although this increase was not statistically significant (86).…”