The fragility of phase 3 trials supporting FDA-approved anticancer medicines: a retrospective analysis

Paggio, Joseph C Del; Tannock, Ian F.

doi:10.1016/s1470-2045(19)30338-9

Cited by 60 publications

(71 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…et al that found a median FI of 2 [9]. The results of Gautret et al are as robust, statistically, as 50% of published trials [12]. Also of note is the lack of systematic RT-PCR testing between the groups consequently time bias.…”

Section: Discussionmentioning

confidence: 87%

How robust are the results of one of the first positive trials exploring hydroxychloroquine for treatment of COVID-19?

Chow

Elsayed

Lock

2020

Preprint

View full text Add to dashboard Cite

An outbreak of a novel human coronavirus infection emerged in Wuhan, China in December 2019. Two months later, the World Health Organization (WHO) announced SARS-CoV-2 as the name for the new virus and COVID-19 for the associated illness. On March 12, 2020, the WHO officially declared COVID-19 as a pandemic. The scientific community has raced to find effective therapeutic agents against the virus. Gautret et al 2020 is among one of the first purportedly positive trials of hydroxychloroquine for the treatment of COVID-19. However, it is imperative that a thorough analysis and understanding of trial data be undertaken prior to making claims about safety and efficacy. Our group assessed the statistical robustness of the trial using the Fragility Index (FI). The FI provides a numerical quantification of a clinical trial's conclusions. The index is based on iterative statistical calculations to determine the minimum number of events within a trial that would theoretically need to change from positive to negative in order for the trial's endpoint to convert from significant to non-significant; the higher the index, the more statistically robust the study results. For the Gautret et al trial, one endpoint had an FI of 1, two had indices of 2, and another had an index of 4. The primary endpoint of viral clearance on day 6 had an FI of 4. This indicates that if 4 events were to change from positive to negative, the conclusion of the trial would become mathematically non-significant. This index is comparable to many other published trials of established agents; the median FI across the reported literature appears to be 2. In conclusion, the trial results reported by Gautret et al are statistically robust, assuming that data quality is not compromised; however, the study was an open-label trial with non-homogenous groups, with analysis conducted per-protocol.Additionally, SARS-CoV-2 Reverse Transcriptase-PCR (RT-PCR) testing was not conducted in a systematic way amongst the two groups. Further analyses of this trial and future trials of All rights reserved. No reuse allowed without permission.(which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

show abstract

Section: Discussionmentioning

confidence: 87%

How robust are the results of one of the first positive trials exploring hydroxychloroquine for treatment of COVID-19?

Chow

Elsayed

Lock

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…One paper applying the FI to oncology studies notes that, while the log-rank test is superior to Fisher's exact test in incorporating time-to-event information into testing, it relies on the proportional hazards assumption, which is not needed by Fisher's exact test. 2 However, violation of the proportional hazards assumption does not inflate type I error; it causes loss of power. Since the FI is applied to assess robustness of significant results, a type I error is the only type of error relevant to its application.…”

Section: How Well Does the Fi Quantify Robustness To Violations Of Momentioning

confidence: 99%

“…The FI has been applied in review studies of anticancer medicines, clinical heart failure, anesthesiology, and several other areas of study, in order to characterize robustness of results amid concern over reproducibility of research. [2][3][4][5][6][7][8][9][10][11][12][13][14] The Fragility Index has been criticized for using Fisher's exact test as its basis when many trials use survival analysis, which is more powerful than Fisher's exact test and can demonstrate differences in time to event as well as the probability of an event. One review of oncology studies found several trials with P-values <.001 and FI values of zero (implying a P-value >.05 by Fisher's exact test).…”

Section: Introductionmentioning

confidence: 99%

“…One review of oncology studies found several trials with P-values <.001 and FI values of zero (implying a P-value >.05 by Fisher's exact test). 2 When the data were reanalyzed with a modified FI that took into account the survival times rather than just the numbers of events, the fragility decreased. 15 In other words, the FI exaggerates fragility of trials whose primary analysis is time-to-event, a point which has been acknowledged by its proponents.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dismantling the Fragility Index: A demonstration of statistical reasoning

Potter

2020

Statistics in Medicine

View full text Add to dashboard Cite

The Fragility Index has been introduced as a complement to the P-value to summarize the statistical strength of evidence for a trial's result. The Fragility Index (FI) is defined in trials with two equal treatment group sizes, with a dichotomous or time-to-event outcome, and is calculated as the minimum number of conversions from nonevent to event in the treatment group needed to shift the P-value from Fisher's exact test over the .05 threshold. As the index lacks a well-defined probability motivation, its interpretation is challenging for consumers. We clarify what the FI may be capturing by separately considering two scenarios: (a) what the FI is capturing mathematically when the probability model is correct and (b) how well the FI captures violations of probability model assumptions. By calculating the posterior probability of a treatment effect, we show that when the probability model is correct, the FI inappropriately penalizes small trials for using fewer events than larger trials to achieve the same significance level. The analysis shows that for experiments conducted without bias, the FI promotes an incorrect intuition of probability, which has not been noted elsewhere and must be dispelled. We illustrate shortcomings of the FI's ability to quantify departures from model assumptions and contextualize the FI concept within current debate around the null hypothesis significance testing paradigm. Altogether, the FI creates more confusion than it resolves and does not promote statistical thinking. We recommend against its use. Instead, sensitivity analyses are recommended to quantify and communicate robustness of trial results.

show abstract

“… 14 , 15 The fragility of statistical inference can be signified by the ease with which a significant P value ( P < .05) crosses over the significance threshold ( P > .05). 16 , 17 Johnson et al 18 introduced a method to compute the fragility for survival analysis by iteratively adding artificial patients to the experimental group with events at the mean exposure time of all individuals until significance is lost. Using this method, one study has recently shown that the fragility index of time-to-event data can be used to estimate the level of confidence of positive results reported in randomized clinical trials (RCTs) leading to FDA approval of anticancer drugs.…”

Section: Introductionmentioning

confidence: 99%

Survival-Inferred Fragility Index of Phase 3 Clinical Trials Evaluating Immune Checkpoint Inhibitors

et al. 2020

View full text Add to dashboard Cite

Key Points Question How stable are the conclusions of phase 3 randomized clinical trials of immune checkpoint inhibitors in oncology? Findings This cross-sectional study of 45 randomized clinical trials calculated the survival-inferred fragility index and found that many oncologic trials assessing immune checkpoint inhibitors have a low survival-inferred fragility index, often less than a small fraction of the sample size and less than the number of patients censored soon after randomization. Meaning These results challenge the robustness of many phase 3 randomized clinical trials of immune checkpoint inhibitors in oncology and address the uncertainty regarding their potential clinical benefit.

show abstract

The fragility of phase 3 trials supporting FDA-approved anticancer medicines: a retrospective analysis

Cited by 60 publications

References 32 publications

How robust are the results of one of the first positive trials exploring hydroxychloroquine for treatment of COVID-19?

How robust are the results of one of the first positive trials exploring hydroxychloroquine for treatment of COVID-19?

Dismantling the Fragility Index: A demonstration of statistical reasoning

Survival-Inferred Fragility Index of Phase 3 Clinical Trials Evaluating Immune Checkpoint Inhibitors

Contact Info

Product

Resources

About