The Fragility of Statistical Significance in Cartilage Restoration of the Knee: A Systematic Review of Randomized Controlled Trials

Abstract

Objective The purpose of this study was to utilize fragility analysis to assess the robustness of randomized controlled trials (RCTs) evaluating the management of articular cartilage defects of the knee. We hypothesize that the cartilage restorative literature will be fragile with the reversal of only a few outcome events required to change statistical significance. Design RCTs from 11 orthopedic journals indexed on PubMed from 2000 to 2020 reporting dichotomous outcome measures relating to the management of articular cartilage defects of the knee were included. The Fragility Index (FI) for each outcome was calculated through the iterative reversal of a single outcome event until significance was reversed. The Fragility Quotient (FQ) was calculated by dividing each FI by study sample size. Additional statistical analysis was performed to provide median FI and FQ across subgroups. Results Nineteen RCTs containing 60 dichotomous outcomes were included for analysis. The FI and FQ of all outcomes was 4 (IQR 2-7) and 0.067 (IQR 0.034-0.096), respectively. The average number of patients lost to follow-up (LTF) was 3.9 patients with 15.8% of the included studies reporting LTF greater than or equal to 4, the FI of all included outcomes. Conclusions The orthopedic literature evaluating articular cartilage defects of the knee is fragile as the reversal of relatively few outcome events may alter the significance of statistical findings. We therefore recommend comprehensive fragility analysis and triple reporting of the P value, FI, and FQ to aid in the interpretation and contextualization of clinical findings reported in the cartilage restoration literature.

The Statistical Fragility of Operative vs Nonoperative Management for Achilles Tendon Rupture: A Systematic Review of Comparative Studies

“…These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 10–14,17,18,21,23,29 –31,36,38,42,47 –53,55 –57 and FQ of 0.048. 10–12,18,21,…”

“…For the outcomes included in this analysis, the median FI was 2.5 (IQR 2-4), the mean FI was 2.9 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 [10][11][12][13][14]17,18,21,23,[29][30][31]36,38,42,[47][48][49][50][51][52][53][55][56][57] and FQ of 0.048. [10][11][12]18,21,23,31,[47][48][49][50][51]53 The hypothesis of this study was confirmed, with more than half (78%) of outcomes analyzed having a loss to follow-up greater than the fragility index for that outcome.…”

“…These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 [10][11][12][13][14]17,18,21,23,[29][30][31]36,38,42,[47][48][49][50][51][52][53][55][56][57] and FQ of 0.048. [10][11][12]18,21,23,31,[47][48][49][50][51]53 The hypothesis of this study was confirmed, with more than half (78%) of outcomes analyzed having a loss to follow-up greater than the fragility index for that outcome. Furthermore, outcomes from studies with a greater risk of bias and rerupture as an individual outcome were more fragile than the other outcomes observed in this analysis.…”

The Fragility of Significance in the Hip Arthroscopy Literature

“…Furthermore, in an evaluation of the orthopaedic oncology literature, Forrester et al found that 60% of the outcomes had an FI value that was less than or equal to the LTF value 23 . Parisien et al, in a recent fragility analysis evaluating the cartilage restoration literature, found that 15.8% of studies either did not report LTF data or reported an LTF value that was greater than the FI 29 . Additionally, in a systematic review and meta-analysis of RCTs evaluating the use of platelet-rich plasma in rotator cuff surgery, Parisien et al revealed that, of the studies reporting LTF data, 30.2% reported an LTF value that was greater than the FI 30 .…”

Statistical Fragility of Single-Row Versus Double-Row Anchoring for Rotator Cuff Repair: A Systematic Review of Comparative Studies

“…For this review, the median FI was found to be 2, and the median FQ was 0.035, which is consistent with previous orthopaedic literature reporting an average median FI of 3.67 ¶ and FQ of 0.036. 9 , 19 , 27 , 35 – 37 , 45 This study’s findings demonstrated that statistical significance can be changed with <3 outcome reversals, or reversal of 3.4% of outcome events. Furthermore, 13 of the 16 significant outcomes (81%) reported needed just a single outcome reversal to become statistically nonsignificant.…”