Heterogeneity in application, design, and analysis characteristics was found for controlled before-after and interrupted time series studies included in Cochrane reviews

Pieper, Dawid; Burns, Jacob; Fretheim, Atle; Ramsay, Craig; Higgins, Julian P T; Mathes, Tim; Pfadenhauer, Lisa M.; Rehfuess, Eva

doi:10.1016/j.jclinepi.2017.07.008

Cited by 38 publications

(43 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…25 163 Consistent with previous systematic reviews, this review highlights variations in methodological considerations (or application) and reporting of ITS. [19][20][21] For example, 20 They also found that key methodological considerations in ITS including seasonality and non-stationarity were considered in less than half of studies. 20 However, autocorrelation was considered in two-thirds of studies included in Jandoc et al's review unlike in our review where it was considered in only about half of studies.…”

Section: Discussionmentioning

confidence: 99%

Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review

et al. 2020

View full text Add to dashboard Cite

BackgroundWhen randomisation is not possible, interrupted time series (ITS) design has increasingly been advocated as a more robust design to evaluating health system quality improvement (QI) interventions given its ability to control for common biases in healthcare QI. However, there is a potential risk of producing misleading results when this rather robust design is not used appropriately. We performed a methodological systematic review of the literature to investigate the extent to which the use of ITS has followed best practice standards and recommendations in the evaluation of QI interventions.MethodsWe searched multiple databases from inception to June 2018 to identify QI intervention studies that were evaluated using ITS. There was no restriction on date, language and participants. Data were synthesised narratively using appropriate descriptive statistics. The risk of bias for ITS studies was assessed using the Cochrane Effective Practice and Organisation of Care standard criteria. The systematic review protocol was registered in PROSPERO (registration number: CRD42018094427).ResultsOf 4061 potential studies and 2028 unique records screened for inclusion, 120 eligible studies assessed eight QI strategies and were from 25 countries. Most studies were published since 2010 (86.7%), reported data using monthly interval (71.4%), used ITS without a control (81%) and modelled data using segmented regression (62.5%). Autocorrelation was considered in 55% of studies, seasonality in 20.8% and non-stationarity in 8.3%. Only 49.2% of studies specified the ITS impact model. The risk of bias was high or very high in 72.5% of included studies and did not change significantly over time.ConclusionsThe use of ITS in the evaluation of health system QI interventions has increased considerably over the past decade. However, variations in methodological considerations and reporting of ITS in QI remain a concern, warranting a need to develop and reinforce formal reporting guidelines to improve its application in the evaluation of health system QI interventions.

show abstract

Section: Discussionmentioning

confidence: 99%

Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review

et al. 2020

View full text Add to dashboard Cite

show abstract

“…While the study will be limited by our ability to identify all potentially eligible reviews and meta-analyses of ITS studies, our search strategy attempts to capture the various ways these studies are described. However, given ITS studies are often not identified as such 16 and that our search is restricted to articles written in English, it is likely that we will not capture all reviews and meta-analyses that include ITS studies. Conversely, we may end up including reviews where no information regarding the definition of the included ITS studies is provided, or where an inappropriate label of ITS has been applied to included studies.…”

Section: Discussionmentioning

confidence: 99%

“…Reviews of ITS studies will be identified using terms adapted from the search strategies of these publications and then combined with terms to identify meta-analyses and systematic reviews. As there is little consistency in the terminology used to describe ITS studies 15 , 16 , our search terms are intentionally broad to achieve greater search sensitivity. Terms will be searched both as free text in the titles, abstracts and keywords fields, and as MeSH terms (or equivalent) where applicable.…”

Section: Methodsmentioning

confidence: 99%

“…For two-stage meta-analysis, a notable challenge is that many primary ITS studies are analysed incorrectly. For example, ITS studies may be analysed as though the study was a before-after design 13 , or analysed as an ITS design, but without taking account of the correlation between observations over time (known as autocorrelation) 14 – 16 . The former is likely to result in estimates of the effect of the interruption that are biased, while the latter is likely to result in estimates of standard errors that are too small 17 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Methods used to meta-analyse results from interrupted time series studies: A methodological systematic review protocol

et al. 2020

View full text Add to dashboard Cite

Background: Systematic reviews are used to inform healthcare decision making. In reviews that aim to examine the effects of organisational, policy change or public health interventions, or exposures, evidence from interrupted time series (ITS) studies may be included. A core component of many systematic reviews is meta-analysis, which is the statistical synthesis of results across studies. There is currently a lack of guidance informing the choice of meta-analysis methods for combining results from ITS studies, and there have been no studies examining the meta-analysis methods used in practice. This study therefore aims to describe current meta-analysis methods used in a cohort of reviews of ITS studies. Methods: We will identify the 100 most recent reviews (published between 1 January 2000 and 11 October 2019) that include meta-analyses of ITS studies from a search of eight electronic databases covering several disciplines (public health, psychology, education, economics). Study selection will be undertaken independently by two authors. Data extraction will be undertaken by one author, and for a random sample of the reviews, two authors. From eligible reviews we will extract details at the review level including discipline, type of interruption and any tools used to assess the risk of bias / methodological quality of included ITS studies; at the meta-analytic level we will extract type of outcome, effect measure(s), meta-analytic methods, and any methods used to re-analyse the individual ITS studies. Descriptive statistics will be used to summarise the data. Conclusions: This review will describe the methods used to meta-analyse results from ITS studies. Results from this review will inform future methods research examining how different meta-analysis methods perform, and ultimately, the development of guidance.

show abstract

“…(8) A careful analysis of data from a before and after study, therefore, should focus on assessing whether an intervention has an effect beyond that due to any pre-existing trends. (9) Why is this important?…”

Section: Pitfalls and Dangers Of Before And After Studiesmentioning

confidence: 99%

Reflection on research methods: The before and after study.

Cheik-Hussein

Harris

Lewin

2019

Preprint

View full text Add to dashboard Cite

Background Before and after studies allow for the investigation of population-level health interventions and are a valuable study design in situations where randomisation is not feasible. The before and after study design involves measuring an outcome both before and after an intervention and comparing the outcome rates in both time periods to determine the effectiveness of the intervention. These studies do not involve a contemporaneous control group and must therefore take into account any underlying secular trends in order to separate the effect of the intervention from any pre-existing trend. Neglecting this important step can lead to spurious results.Methods To illustrate the importance of accounting for underlying trends, we performed a before and after study assessing 30-day mortality in hip fracture patients without any actual intervention, and instead designated an arbitrarily-chosen time-point as our ‘intervention’. We did this to ensure that we were basing our results exclusively on the underlying trend throughout the studied period and also to enable us to show that even an intervention of nothing may be spuriously interpreted to have an effect if the before and after study is incorrectly analysed. Results We found a secular trend in our data showing improving 30-day mortality in hip fracture patients in our institution. We then demonstrated that disregarding this underlying trend showed that our intervention of nothing ‘resulted’ in a significant decrease in mortality, from 6.7% in the ‘before’ period to 3.1% in the ‘after’ period (p<0.0008). This apparent impact on mortality disappeared when we accounted for the underlying trend in our analysis (IRR of 0.75, 95% CI 0.32 – 1.78; p=0.5). In the context of declining 30-day mortality following hip fracture, failure to consider the existing underlying trend lead us to believe that it was our ‘intervention’ that ‘caused’ the decrease in mortality in the ’after’ period compared to the ‘before’ period when our results clearly show that mortality was decreasing irrespective of any intervention.Conclusion Our study highlights the importance of appropriate measurement and consideration of underlying trends when analysing data from before and after studies and illustrates what can happen should researchers neglect this important step.

show abstract

Heterogeneity in application, design, and analysis characteristics was found for controlled before-after and interrupted time series studies included in Cochrane reviews

Cited by 38 publications

References 72 publications

Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review

Use of interrupted time series methods in the evaluation of health system quality improvement interventions: a methodological systematic review

Methods used to meta-analyse results from interrupted time series studies: A methodological systematic review protocol

Reflection on research methods: The before and after study.

Contact Info

Product

Resources

About