A systematic survey of randomised trials that stopped early for reasons of futility

Abstract

Background: Randomised trial protocols may incorporate interim analyses, with the potential to stop the study for futility if early data show insufficient promise of a treatment benefit. Previously, we have shown that this approach will theoretically lead to mis-estimation of the treatment effect. We now wished to ascertain the importance of this phenomenon in practice. Methods: We reviewed the methods and results in a set of trials that had stopped for futility, identified through an extensive literature search. We recorded clinical areas, interventions, study design, outcomes, trial setting, sponsorship, planned and actual treatment effects, sample sizes; power; and if there was a data safety monitoring board, or a published protocol. We identified: if interim analyses were pre-specified, and how many analyses actually occurred; what pre-specified criteria might define futility; if a futility analysis formed the basis for stopping; who made the decision to stop; and the conditional power of each study, i.e. the probability of statistically significant results if the study were to continue to its complete sample size. Results: We identified 52 eligible trials, covering many clinical areas. Most trials had multiple centres, tested drugs, and 40% were industry sponsored. There were 75% where at least one interim analysis was planned a priori; a majority had only one interim analysis, typically with about half the target total sample size. A majority of trials did not pre-define a stopping rule, and a variety of reasons were given for stopping. Few studies calculated and reported low conditional power to justify the early stop. When conditional power could be calculated, it was typically low, especially under the current trend hypothesis. However, under the original design hypothesis, a few studies had relatively high conditional power. Data collection often continued after the interim analysis. Conclusions: Although other factors will typically be involved, we conclude that, from the perspective of conditional power, stopping early for futility was probably reasonable in most cases, but documentation of the basis for stopping was often missing or vague. Interpretation of truncated trials would be enhanced by improved reporting of stopping protocols, and of their actual execution.

A multi-center phase II randomized clinical trial of losartan on symptomatic outpatients with COVID-19

“…Critically, due to limitations in funding and a focus on a pragmatic design, we were not able to assess the effect of the intervention on the RAAS pathways in question, either systemically or in the lungs, though this remains an area of further investigation. Finally, we acknowledge that studies stopped early for futility have been an area of debate for 40 years [41] . We made the difficult decision of stopping early because of the observed hospitalization rates, logistical challenges, and concern of more equitable allocation of potential study participants into larger national outpatient trial efforts more likely to identify an effective therapy.…”

The efficacy and safety of hydroxychloroquine for COVID-19 prophylaxis: A systematic review and meta-analysis of randomized trials

“…To our knowledge, this is the first systematic review on this novel topic. There are several strengths of this review, we adhered to a rigorous process with an extensive systematic search of the literature, summary of current trial registries, duplication of all aspects of the review, and adherence to PRISMA guidelines ( S11 Table ) [ 38 ].…”

Conditional power of antidepressant network meta-analysis

“…Results were reported in terms of two conditional power indices quantifying sample sizes: N CP = 20 % : Sample size at 80% conditional power, which conventionally implies that a trial investigating a true effect will correctly reject the null hypothesis 80% of the time and will report a false negative (commit a type II error) in the remaining 20% of cases [ 9 ]. N CP = 80 % : Sample size at 20% conditional power, which conventionally may be regarded as futility boundary with values below indicating that a trial is likely to be futile under the null hypothesis [ 10 ]. …”

“…N CP = 80 % : Sample size at 20% conditional power, which conventionally may be regarded as futility boundary with values below indicating that a trial is likely to be futile under the null hypothesis [ 10 ].…”

“…If conditional power analysis suggests for example at least 80% conditional power, which conventionally implies that trial(s) investigating a true effect will correctly reject the null hypothesis [ 9 ], together with a reasonable required sample size, further research may be promising. Otherwise, if such an analysis suggests for example less than 20% conditional power, which conventionally may be regarded as futility boundary with values below indicating that a trial is likely to be futile under the null hypothesis [ 10 ], then it may be recommended to refrain from further RCTs on a given intervention to save resources.…”