The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

Abstract

I n a randomized controlled trial (RCT) comparing treatments A and B, a null hypothesis (H 0 ) of no difference in a primary outcome of interest is defined. Whether any observed difference is statistically significant has traditionally been based on the P value and the confidence interval (CI). For a century, an arbitrary threshold of 0.05 (1/20) has been used to define statistical significance. 1 Because this probability is quite low, we conclude that P ≤ 0.05 suggests that the observed difference between A and B is incompatible with H 0 , and, with a ≥ 95% degree of certainty, that A and B are different. The 95% CI is a lower-upper limit in which the true effect estimate lies within a 95% certainty. A 95% CI that does not include the null effect size indicates the observed difference has reached statistical significance, and H 0 is rejected. The 95% CI provides information about the precision of the estimate and complements the P value. Reporting both is recommended.Use of a P value of 0.05 to dichotomize whether treatments A and B are "truly" different is appealing because of its simplicity. Indeed, for all its many limitations, 2 its use actually increased from 1990 to 2015 in MEDLINE and PubMed Central abstracts and articles. 3 Statistical significance carries considerable weight. Researchers, editors and reviewers, readers, and the press tend to become more excited about positive results. 4 However, a shocking number of scientific studies and meta-analyses are not reproducible or replicable. [5][6][7] A reasonable question to ask is: if a positive study was to be repeated, how "easy" might it be for the results to change from being statistically significant to non-significant (and thus, rightly or wrongly, lose some of their appeal)? What if there was, by sheer chance, one or several more (or less) outcome event(s) in one of the comparative groups? Would the coveted statistical significance be lost? In recent years, there has been much interest in a metric-the fragility index (FI), [8][9][10][11] first proposed 3 decades ago 12 -to test the robustness/fragility of statistically significant results. The various applications and implications of the FI are discussed herein.

The Statistical Fragility of Operative vs Nonoperative Management for Achilles Tendon Rupture: A Systematic Review of Comparative Studies

“…For the outcomes included in this analysis, the median FI was 2.5 (IQR 2-4), the mean FI was 2.9 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 10 – 14 , 17 , 18 , 21 , 23 , 29 – 31 , 36 , 38 , 42 , 47 – 53 , 55 …”

“…These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 10–14,17,18,21,23,29 –31,36,38,42,47 –53,55 –57 and FQ of 0.048. 10–12,18,21,…”

“… 9 Indeed, the fragility index is an offshoot of the P value and therefore should be taken as a metric to aide in the interpretation of the P value. 22 Other important metrics of a study’s robustness such as study design, prospective sample size calculations, preregistration of planned analyses, and transparent reporting of procedures and statistical analyses should all be taken into consideration when interpreting the results of a study. The inclusion of FI and FQ in a given analysis should be viewed as an additional tool in the clinician’s arsenal for the interpretation of the statistical conclusions of a study.…”

The Statistical Fragility of Platelet-Rich Plasma as Treatment for Chronic Noninsertional Achilles Tendinopathy: A Systematic Review and Meta-analysis

“…To our knowledge, only 2 prior studies have written about the CFI and CFQ. Caldwell et al 5 reported a CFI of 9 for 39 nondichotomous outcomes in the sports medicine and arthroscopy literature, and Ho et al 15 calculated a CFI of 3 for a single RCT investigating vagal nerve electrical stimulation after stroke. The present study is thereby the first to thoroughly assess statistical fragility for a specific orthopaedic intervention.…”

Prone positioning might reduce the need for intubation in people with severe COVID-19 – Authors' reply

“… 3 Although fragility index is an increasingly popular metric to allegedly show the robustness of trial results, it has also been referred to as a p value in disguise and a potentially misleading metric. 4 In the hypothetical case regarding patients who withdrew consent, McGuire and colleagues could have calculated that 226 events in 567 patients in the awake prone positioning group and 257 events in 559 patients in the control group yields a p value of 0·041 with a Fisher's exact test. We must stress that the robustness of a study relies on its design, study implementation, analysis plan, effect sizes, CIs, generalisability, and limitations, and not on a single integer derived from an inappropriate secondary analysis of a hypothetical randomised trial that never occurred.…”