Multicentre randomised trials in anaesthesia: an analysis using Bayesian metrics

Abstract

Summary Are the results of randomised trials reliable and are p values and confidence intervals the best way of quantifying efficacy? Low power is common in medical research, which reduces the probability of obtaining a ‘significant result’ and declaring the intervention had an effect. Metrics derived from Bayesian methods may provide an insight into trial data unavailable from p values and confidence intervals. We did a structured review of multicentre trials in anaesthesia that were published in the New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Journal of Anaesthesia and Anesthesiology between February 2011 and November 2021. We documented whether trials declared a non‐zero effect by an intervention on the primary outcome. We documented the expected and observed effect sizes. We calculated a Bayes factor from the published trial data indicating the probability of the data under the null hypothesis of zero effect relative to the alternative hypothesis of a non‐zero effect. We used the Bayes factor to calculate the post‐test probability of zero effect for the intervention (having assumed 50% belief in zero effect before the trial). We contacted all authors to estimate the costs of running the trials. The median (IQR [range]) hypothesised and observed absolute effect sizes were 7% (3–13% [0–25%]) vs. 2% (1–7% [0–24%]), respectively. Non‐zero effects were declared for 12/56 outcomes (21%). The Bayes factor favouring a zero effect relative to a non‐zero effect for these 12 trials was 0.000001–1.9, with post‐test zero effect probabilities for the intervention of 0.0001–65%. The other 44 trials did not declare non‐zero effects, with Bayes factors favouring zero effect of 1–688, and post‐test probabilities of zero effect of 53–99%. The median (IQR [range]) study costs reported by 20 corresponding authors in US$ were $1,425,669 ($514,766–$2,526,807 [$120,758–$24,763,921]). We think that inadequate power and mortality as an outcome are why few trials declared non‐zero effects. Bayes factors and post‐test probabilities provide a useful insight into trial results, particularly when p values approximate the significance threshold.

Sugammadex and neostigmine: when better may not be best

“…Few large multicentre trials in anaesthesia report significant results [11]. The reasons for this are complex but possibly reflect a combination of low achieved power and low trial participant susceptibility to the measured outcome [12].…”

The credibility plot for extreme explanations and all explanations in between

“…The Bayes factor metric calculated by Seretny et al. [4]. The calculation is the credibility that individualised blood pressure management has precisely zero effect on the rate of postoperative organ dysfunction (a relative rate of 100%, white gap, right dashed line) after the trial (red point on grey line) vs. before the trial (red point on dashed thick faint red curve).…”

“…Dashed vertical lines are relative rates of 50% (left) and 100% (right). The red point is the single Bayes factor reported by Seretny et al [4]. Confident belief before the trial that individualised blood pressure management would halve organ dysfunction, from 40% to 20%, plots Bayes factors that are thinner lines, whilst thicker lines are Bayes factors for less certainty in rates before the trial.…”

The credibility plot for extreme explanations and all explanations in between: a reply

“…I did not intend to be dismissive of the paper by Seretny et al. [3]. I am sorry and wish that I had phrased my comments differently.…”