Statistical Multiplicity in Systematic Reviews of Anaesthesia Interventions: A Quantification and Comparison between Cochrane and Non-Cochrane Reviews

Imberger, Georgina; Vejlby, Alexandra Damgaard; Hansen, Signe E J; Møller, Ann Merete; Wetterslev, Jørn

doi:10.1371/journal.pone.0028422

Cited by 39 publications

(32 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Problems with multiplicity has major implications for the interpretation of the confidence interval and the P -value and this is one reason why it should be mandatory to report a predefined outcome hierarchy including a clear definition of a primary outcome before conducting a randomised clinical trial [17,40,46]. The conclusion about trial intervention effects should always be related to the result on the primary outcome (or outcomes) limiting the risk of falsely declaring a trial intervention for being effective.…”

Section: Methods and Resultsmentioning

confidence: 99%

The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials

Jakobsen

Gluud

Winkel

et al. 2014

BMC Med Res Methodol

Self Cite

161

190

View full text Add to dashboard Cite

BackgroundThresholds for statistical significance are insufficiently demonstrated by 95% confidence intervals or P-values when assessing results from randomised clinical trials. First, a P-value only shows the probability of getting a result assuming that the null hypothesis is true and does not reflect the probability of getting a result assuming an alternative hypothesis to the null hypothesis is true. Second, a confidence interval or a P-value showing significance may be caused by multiplicity. Third, statistical significance does not necessarily result in clinical significance. Therefore, assessment of intervention effects in randomised clinical trials deserves more rigour in order to become more valid.MethodsSeveral methodologies for assessing the statistical and clinical significance of intervention effects in randomised clinical trials were considered. Balancing simplicity and comprehensiveness, a simple five-step procedure was developed.ResultsFor a more valid assessment of results from a randomised clinical trial we propose the following five-steps: (1) report the confidence intervals and the exact P-values; (2) report Bayes factor for the primary outcome, being the ratio of the probability that a given trial result is compatible with a ‘null’ effect (corresponding to the P-value) divided by the probability that the trial result is compatible with the intervention effect hypothesised in the sample size calculation; (3) adjust the confidence intervals and the statistical significance threshold if the trial is stopped early or if interim analyses have been conducted; (4) adjust the confidence intervals and the P-values for multiplicity due to number of outcome comparisons; and (5) assess clinical significance of the trial results.ConclusionsIf the proposed five-step procedure is followed, this may increase the validity of assessments of intervention effects in randomised clinical trials.

show abstract

Section: Methods and Resultsmentioning

confidence: 99%

The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials

Jakobsen

Gluud

Winkel

et al. 2014

BMC Med Res Methodol

Self Cite

161

190

View full text Add to dashboard Cite

show abstract

“…As the tests on the accumulating trial population are not statistically independent, the Bonferroni-adjusted levels of statistical significance are most often too conservative [24]. The trial participants in an early sequential analysis are also included in the subsequent later sequential analyses.…”

Section: Introductionmentioning

confidence: 99%

Trial Sequential Analysis in systematic reviews with meta-analysis

Wetterslev

Jakobsen

Gluud

2017

BMC Med Res Methodol

Self Cite

826

558

View full text Add to dashboard Cite

BackgroundMost meta-analyses in systematic reviews, including Cochrane ones, do not have sufficient statistical power to detect or refute even large intervention effects. This is why a meta-analysis ought to be regarded as an interim analysis on its way towards a required information size. The results of the meta-analyses should relate the total number of randomised participants to the estimated required meta-analytic information size accounting for statistical diversity. When the number of participants and the corresponding number of trials in a meta-analysis are insufficient, the use of the traditional 95% confidence interval or the 5% statistical significance threshold will lead to too many false positive conclusions (type I errors) and too many false negative conclusions (type II errors).MethodsWe developed a methodology for interpreting meta-analysis results, using generally accepted, valid evidence on how to adjust thresholds for significance in randomised clinical trials when the required sample size has not been reached.ResultsThe Lan-DeMets trial sequential monitoring boundaries in Trial Sequential Analysis offer adjusted confidence intervals and restricted thresholds for statistical significance when the diversity-adjusted required information size and the corresponding number of required trials for the meta-analysis have not been reached. Trial Sequential Analysis provides a frequentistic approach to control both type I and type II errors. We define the required information size and the corresponding number of required trials in a meta-analysis and the diversity (D2) measure of heterogeneity. We explain the reasons for using Trial Sequential Analysis of meta-analysis when the actual information size fails to reach the required information size. We present examples drawn from traditional meta-analyses using unadjusted naïve 95% confidence intervals and 5% thresholds for statistical significance. Spurious conclusions in systematic reviews with traditional meta-analyses can be reduced using Trial Sequential Analysis. Several empirical studies have demonstrated that the Trial Sequential Analysis provides better control of type I errors and of type II errors than the traditional naïve meta-analysis.ConclusionsTrial Sequential Analysis represents analysis of meta-analytic data, with transparent assumptions, and better control of type I and type II errors than the traditional meta-analysis using naïve unadjusted confidence intervals.

show abstract

“…In that way, the method is used to test a focused hypothesis with a powerful meta-analytic strategy [15]–[17]. One of the advantages of Bayesian MTC methods over conventional meta-analysis is that one can examine and take into consideration the likelihood of the alternative hypothesis (i.e., estimate the probability that a treatment should be preferred).…”

Section: Discussionmentioning

confidence: 99%

Efficacy of New Generation Antidepressants: Differences Seem Illusory

Spielmans

Flückiger

et al. 2013

PLoS ONE

View full text Add to dashboard Cite

BackgroundRecently, Cipriani and colleagues examined the relative efficacy of 12 new-generation antidepressants on major depression using network meta-analytic methods. They found that some of these medications outperformed others in patient response to treatment. However, several methodological criticisms have been raised about network meta-analysis and Cipriani's analysis in particular which creates the concern that the stated superiority of some antidepressants relative to others may be unwarranted.Materials and MethodsA Monte Carlo simulation was conducted which involved replicating Cipriani's network meta-analysis under the null hypothesis (i.e., no true differences between antidepressants). The following simulation strategy was implemented: (1) 1000 simulations were generated under the null hypothesis (i.e., under the assumption that there were no differences among the 12 antidepressants), (2) each of the 1000 simulations were network meta-analyzed, and (3) the total number of false positive results from the network meta-analyses were calculated.FindingsGreater than 7 times out of 10, the network meta-analysis resulted in one or more comparisons that indicated the superiority of at least one antidepressant when no such true differences among them existed.InterpretationBased on our simulation study, the results indicated that under identical conditions to those of the 117 RCTs with 236 treatment arms contained in Cipriani et al.'s meta-analysis, one or more false claims about the relative efficacy of antidepressants will be made over 70% of the time. As others have shown as well, there is little evidence in these trials that any antidepressant is more effective than another. The tendency of network meta-analyses to generate false positive results should be considered when conducting multiple comparison analyses.

show abstract

Statistical Multiplicity in Systematic Reviews of Anaesthesia Interventions: A Quantification and Comparison between Cochrane and Non-Cochrane Reviews

Cited by 39 publications

References 31 publications

The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials

The thresholds for statistical and clinical significance – a five-step procedure for evaluation of intervention effects in randomised clinical trials

Trial Sequential Analysis in systematic reviews with meta-analysis

Efficacy of New Generation Antidepressants: Differences Seem Illusory

Contact Info

Product

Resources

About