The Validity and Precision of the Comparative Interrupted Time-Series Design

Abstract

We explore the conditions under which short, comparative interrupted time-series (CITS) designs represent valid alternatives to randomized experiments in educational evaluations. To do so, we conduct three within-study comparisons, each of which uses a unique data set to test the validity of the CITS design by comparing its causal estimates to those from a randomized controlled trial (RCT) that shares the same treatment group. The degree of correspondence between RCT and CITS estimates depends on the observed pretest time trend differences and how they are modeled. Where the trend differences are clear and can be easily modeled, no bias results; where the trend differences are more volatile and cannot be easily modeled, the degree of correspondence is more mixed, and the best results come from matching comparison units on both pretest and demographic covariates.

Association Between State-Mandated Protocolized Sepsis Care and In-hospital Mortality Among Adults With Sepsis

“…Accordingly, rather than specifying a single postimplementation temporal trend, we fit a model with indicators for each postimplementation quarter. 26,27 The preimplementation period was defined as hospital discharge from January 1, 2011, through March 31, 2013, before the filing of the regulations. The postimplementation period was defined as hospital discharge from April 1, 2013, through September 30, 2015.…”

Effectiveness of antimalarial interventions in Nigeria: Evidence from facility‐level longitudinal data

“…Table A.1 reveals that the ITS results are qualitatively similar to the DD results discussed below, demonstrating robustness of the preferred model. Table A.2 shows that DD is preferred to CITS; the diverging trend hypothesis does not hold statistically, leading to overfitting under CITS . An explanation why CITS performs less well here compared to other studies is perhaps the fact that the number of pre‐intervention data points in this study is too small (five data points: March‐July 2010) to properly establish pre‐intervention trends and observe trend divergence …”

“…The purpose of exploring the ITS design in addition to DD is to provide robustness checks. The DD design has indeed been considered the more rigorous approach, as it explicitly incorporates the non‐intervention group, thereby having better degrees of freedom and more generalizability …”

“…The method has been proved to be valid and precise, and is considered more flexible than DD, rendering it superior under certain conditions. More specifically, while the DD method assumes that outcome trends are similar, that is, “common,” between the intervention and the non‐intervention groups, the CITS method relaxes the assumption and allows those trends to diverge . The CITS design is considered more appropriate if trends of the outcomes of interest statistically differ between the intervention and the non‐intervention groups.…”

“…Finally, u ij tackles the problem of autocorrelation, allowing for temporal variations of the outcomes at the facility level. The inclusion of a linear trend and month fixed effects, and the use of mixed‐effects modeling are observed in other policy‐evaluation studies.…”