Trial Sequential Analysis in systematic reviews with meta-analysis

Abstract

BackgroundMost meta-analyses in systematic reviews, including Cochrane ones, do not have sufficient statistical power to detect or refute even large intervention effects. This is why a meta-analysis ought to be regarded as an interim analysis on its way towards a required information size. The results of the meta-analyses should relate the total number of randomised participants to the estimated required meta-analytic information size accounting for statistical diversity. When the number of participants and the corresponding number of trials in a meta-analysis are insufficient, the use of the traditional 95% confidence interval or the 5% statistical significance threshold will lead to too many false positive conclusions (type I errors) and too many false negative conclusions (type II errors).MethodsWe developed a methodology for interpreting meta-analysis results, using generally accepted, valid evidence on how to adjust thresholds for significance in randomised clinical trials when the required sample size has not been reached.ResultsThe Lan-DeMets trial sequential monitoring boundaries in Trial Sequential Analysis offer adjusted confidence intervals and restricted thresholds for statistical significance when the diversity-adjusted required information size and the corresponding number of required trials for the meta-analysis have not been reached. Trial Sequential Analysis provides a frequentistic approach to control both type I and type II errors. We define the required information size and the corresponding number of required trials in a meta-analysis and the diversity (D2) measure of heterogeneity. We explain the reasons for using Trial Sequential Analysis of meta-analysis when the actual information size fails to reach the required information size. We present examples drawn from traditional meta-analyses using unadjusted naïve 95% confidence intervals and 5% thresholds for statistical significance. Spurious conclusions in systematic reviews with traditional meta-analyses can be reduced using Trial Sequential Analysis. Several empirical studies have demonstrated that the Trial Sequential Analysis provides better control of type I errors and of type II errors than the traditional naïve meta-analysis.ConclusionsTrial Sequential Analysis represents analysis of meta-analytic data, with transparent assumptions, and better control of type I and type II errors than the traditional meta-analysis using naïve unadjusted confidence intervals.

Closure, Anticoagulation, or Antiplatelet Therapy for Cryptogenic Stroke With Patent Foramen Ovale: Systematic Review of Randomized Trials, Sequential Meta‐Analysis, and New Insights From the CLOSE Study

“…Our trial sequential analysis provides further support to the argument that this conclusion is not a false‐positive result. Indeed, concluding that a cumulative meta‐analysis is significant at the usual nominal 0.05 threshold may correspond to a false‐positive result when the required information size has not been reached, as in the present study 16. By contrast, the TSA provides an adjusted, more stringent level of statistical significance, in order to limit the risk of type I error 15.…”

“…Indeed, traditional meta‐analytic methods may sometimes lead to false‐positive results, especially when pooled estimates are updated with the publication of new trials in cumulative meta‐analyses 14. The aim of this approach is to determine whether enough evidence has been obtained to reach a conclusion regarding the superiority of one treatment over another, or if a new RCT addressing the same comparison should be undertaken 15, 16. Briefly, in TSA, a cumulative meta‐analysis of RCTs is similar to several interim analyses of a single RCT, with the construction of specific trial sequential monitoring boundaries by using the Lan–DeMets α spending function approach 15, 16.…”

“…The aim of this approach is to determine whether enough evidence has been obtained to reach a conclusion regarding the superiority of one treatment over another, or if a new RCT addressing the same comparison should be undertaken 15, 16. Briefly, in TSA, a cumulative meta‐analysis of RCTs is similar to several interim analyses of a single RCT, with the construction of specific trial sequential monitoring boundaries by using the Lan–DeMets α spending function approach 15, 16. In the present TSA, the low‐bias heterogeneity‐adjusted information size method was used to determine the required information size 26.…”

“…To this aim, we included published data from all RCTs and unpublished data from the 3‐arm CLOSE trial, allowing head‐to‐head comparison of all the abovementioned therapeutic strategies and calculation of the absolute risk of stroke and corresponding number needed to treat. Because traditional updated meta‐analyses may sometimes lead to false‐positive results due to repeated significance testing,14 we performed trial sequential analyses (TSA), a method similar to interim analyses of a single RCT, in order to determine whether enough evidence has been obtained to reach a conclusion 15, 16…”

From the Editors’ Desk: Bias in Systematic Reviews—Let the Reader Beware

“…A recent article discussed the benefits of Trial Sequential Analysis (TSA), a method that allows metaanalysts to determine the likelihood that the review contains either a type I error (finding a difference that does not exist) or a type II error (reporting no difference when in fact one exists). 3 TSA theory is similar to the principles governing stopping rules for interventional trials. It takes the total sample size and the size of the effect, and establishes boundaries.…”