Dismantling the Fragility Index: A demonstration of statistical reasoning

Abstract

The Fragility Index has been introduced as a complement to the P-value to summarize the statistical strength of evidence for a trial's result. The Fragility Index (FI) is defined in trials with two equal treatment group sizes, with a dichotomous or time-to-event outcome, and is calculated as the minimum number of conversions from nonevent to event in the treatment group needed to shift the P-value from Fisher's exact test over the .05 threshold. As the index lacks a well-defined probability motivation, its interpretation is challenging for consumers. We clarify what the FI may be capturing by separately considering two scenarios: (a) what the FI is capturing mathematically when the probability model is correct and (b) how well the FI captures violations of probability model assumptions. By calculating the posterior probability of a treatment effect, we show that when the probability model is correct, the FI inappropriately penalizes small trials for using fewer events than larger trials to achieve the same significance level. The analysis shows that for experiments conducted without bias, the FI promotes an incorrect intuition of probability, which has not been noted elsewhere and must be dispelled. We illustrate shortcomings of the FI's ability to quantify departures from model assumptions and contextualize the FI concept within current debate around the null hypothesis significance testing paradigm. Altogether, the FI creates more confusion than it resolves and does not promote statistical thinking. We recommend against its use. Instead, sensitivity analyses are recommended to quantify and communicate robustness of trial results.

Meta-analyses in paediatric surgery are often fragile: implications and consequences

“…Penalisation of the smaller study is exemplified by the following comparison of two hypothetical studies: if one smaller study revealed a relative risk reduction of 89%, and this is compared to a larger study with a risk reduction of 20%, they may have equal P values of 0.02. In this example, the smaller study would have a fragility index of one compared to a fragility index of nine in the larger study despite a highly different effect size [26]. The fragility index can actively be influenced by the a priori power: the higher this parameter is calculated for, the larger the fragility index will result-in particular if small effect sizes are chased-due to the relationship to sample size [24].…”

Comparative Studies in the Shoulder Literature Lack Statistical Robustness: A Fragility Analysis

“…[7][8][9][10][11][12][13][14][15][16][17][18] However, similar to the P value, the FI is an imperfect measure of trial stability in isolation as it provides an absolute measure of fragility without reference to sample size. Potter 20 has recently criticized the use of a FI in favor of sensitivity analysis to quantify the robustness of trial results. In her analysis, Potter correctly addresses the limitation of the FI in isolation as it may inappropriately penalize small trials FRAGILITY ANALYSIS OF THE SHOULDER LITERATURE e3 for utilizing fewer events but fails to recognize the complementary value of the FQ, which accounts for sample size.…”

Outcome of acute kidney injury: how to make a difference?

“…But frailty index can also be interpreted as the reflection of a consistent choice of the size of the population studied for the size of the effect observed. Finally, its use is recently debated because it has been proven to lack the ability of the frailty index to quantify deviations from a model's null assumptions [56]. While the case of balanced crystalloids vs normal saline is not closed, accumulating evidence strongly suggest that (1) normal saline is not superior to balanced solution and (2) balanced solution are likely to be superior to normal saline in acutely and critically ill patient.…”