Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased

Fortson, Kenneth; Gleason, Philip; Kopa, Emma; Verbitsky-Savitz, Natalya

doi:10.1016/j.econedurev.2014.11.001

Cited by 32 publications

(42 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fewer studies measure mean differences in bias between quasi-experimental and experimental estimates, and compare this to a threshold considered meaningful by policymakers [23,35,78]. Fewer still attempt to measure the statistical significance of differences in effect sizes between designs [10,29,30].…”

Section: Metric For Concordancementioning

confidence: 99%

See 1 more Smart Citation

Testing the Validity of the Single Interrupted Time Series Design

Baicker

Svoronos

2019

SSRN Journal

View full text Add to dashboard Cite

Given the complex relationships between patients' demographics, underlying health needs, and outcomes, establishing the causal effects of health policy and delivery interventions on health outcomes is often empirically challenging. The single interrupted time series (SITS) design has become a popular evaluation method in contexts where a randomized controlled trial is not feasible. In this paper, we formalize the structure and assumptions underlying the single ITS design and show that it is significantly more vulnerable to confounding than is often acknowledged and, as a result, can produce misleading results. We illustrate this empirically using the Oregon Health Insurance Experiment, showing that an evaluation using a single interrupted time series design instead of the randomized controlled trial would have produced large and statistically significant results of the wrong sign. We discuss the pitfalls of the SITS design, and suggest circumstances in which it is and is not likely to be reliable.

show abstract

Section: Metric For Concordancementioning

confidence: 99%

“…In order to determine whether differences between two designs are statistically significant, we use a bootstrapping method used in some of the more rigorous WSCs to date [10,29,30,69]…”

Section: Computing Standard Errorsmentioning

confidence: 99%

Testing the Validity of the Single Interrupted Time Series Design

Baicker

Svoronos

2019

SSRN Journal

View full text Add to dashboard Cite

show abstract

“…A potential limitation of traditional regression is that even if all relevant confounders are controlled for, the model may still yield biased estimates if the relationship between these factors and the outcome measure is not appropriately modelled. This issue may become a great concern in situations where the case and control groups differ essentially in the distribution of background characteristics (Fortson et al 2015). To estimate a population effect while also addressing this issue, we extend the analysis by using Propensity Score Matching (PSM) (Abadie and Imbens 2006).…”

Section: Methodsmentioning

confidence: 99%

“…However, similarly as in the regression approach, PSM cannot account for unmeasured confounders. Both methods, therefore, relied on the assumption that there are no additional factors correlated with both the T1DM and school grades (Fortson et al 2015). 4 The matched case-control design of the data in this study implies not only that T1DM is overrepresented in the sample compared to in the general population but also that the information about the distribution of the matching variables is lost.…”

Section: Methodsmentioning

confidence: 99%

Effect of type 1 diabetes on school performance in a dynamic world: new analysis exploring Swedish register data

Persson

Gerdtham

et al. 2018

Applied Economics

View full text Add to dashboard Cite

This paper investigates if the effect of type 1 diabetes mellitus (T1DM) on school performance, documented in prior research, has changed in more recent birth cohorts of children using national Swedish population register data. The issue is of interest because management and treatment of the disease have improved over the last decades and, furthermore, because of changes in the educational grading system. Despite these changes, data indicate a persistent negative effect of T1DM on compulsory and upper secondary school grades with a standardized effect size of −0.109 and −0.070, respectively, and the results appear only marginally smaller compared to earlier findings in cohorts completing school under the previous grading system. Moreover, the results are consistent for alternative model specifications and econometric estimation strategies. Whereas access to new treatment technologies and improved diabetes management strategies has reduced the burden of diabetes in daily life, the results from this study indicate that continued efforts are needed to improve the situation in school for children with T1DM to prevent potential long-term socioeconomic consequences.

show abstract

“…PSM approach has become popular in the context of program evaluation, which is extensively used by statisticians and economists until recently. Many notable examples include (Heckman et al, 1997(Heckman et al, , 1998Harding, 2003;List et al, 2003;Mensah et al, 2010;Ponzo, 2013;Fortson et al, 2015).…”

Section: Propensity Score Matching Approachmentioning

confidence: 99%