Comparative Studies in the Shoulder Literature Lack Statistical Robustness: A Fragility Analysis

Abstract

Purpose: Evidenced-based decision-making is rooted in comparative clinical studies; however, a small number of outcome event reversals have the potential to change study significance. The purpose of this study was to determine the utility of applying fragility analysis to comparative studies in the published orthopaedic shoulder literature. Methods: Comparative clinical shoulder research studies reporting 1:1 dichotomous categorical data were analyzed in 6 leading orthopaedic journals between 2006 and 2016. Statistical significance was defined as a P value of less than .05. The fragility index (FI) for each study outcome was determined by the number of event reversals required to change the P value to either greater or less than 0.05, thus changing the study conclusions. The associated fragility quotient (FQ) was determined by dividing the FI by the total population comprising a particular outcome. Results: Of the 23,897 studies screened, 3,591 met search criteria, with 198 comparative studies ultimately included for analysis, 67 of which were randomized controlled trials. There were 357 total outcome events with 74 reported as significant and 283 as not significant. The FI was 4 (IQR 2-6) with an associated FQ of 0.066 (interquartile range [IQR] 0.038-0.102). There was no difference in statistical fragility between randomized and nonrandomized trials with both revealing a FI of 4 and FQ of 0.068 (IQR 0.044-0.107) and 0.065 (IQR 0.031-0.101), respectively. Conclusions: This current analysis reveals that comparative shoulder studies published in six leading orthopaedic journals are at risk of statistical fragility. As such, contemporary clinical shoulder literature may not be as robust as traditionally perceived with the reversal of only a few outcome events required to change study significance. Therefore, we advocate the reporting of both FI and FQ in addition to the P value as statistical complements to all comparative investigations to provide a more comprehensive understanding of trial stability and significance in the published shoulder literature. Clinical Relevance: Comparative study designs are commonly employed in shoulder research. Several studies in both the general medical and orthopaedic literature have identified a lack of statistical robustness through comprehensive fragility analysis. Our findings demonstrate the P value may be an inadequate independent statistical metric requiring the complement of a FI and FQ to aid in the interpretation and understanding of study significance for clinical decision-making.T he primary objective of evidence-based medicine is to produce meaningful and clinically relevant information to help guide medical decision-making. 1 The viability of evidence-based medicine depends on validated research findings and an informed readership. Within the shoulder surgery literature, randomized controlled trials (RCTs) and dichotomous comparison studies are frequently used toward this end. 2 Statistical findings are typically reported in terms of a threshold probability value (P ...

The Statistical Fragility of Operative vs Nonoperative Management for Achilles Tendon Rupture: A Systematic Review of Comparative Studies

“…For the outcomes included in this analysis, the median FI was 2.5 (IQR 2-4), the mean FI was 2.9 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 10 – 14 , 17 , 18 , 21 , 23 , 29 – 31 , 36 , 38 , 42 , 47 – 53 , 55 …”

“…These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 10–14,17,18,21,23,29 –31,36,38,42,47 –53,55 –57 and FQ of 0.048. 10–12,18,21,…”

“…For the outcomes included in this analysis, the median FI was 2.5 (IQR 2-4), the mean FI was 2.9 (±1.58), the median FQ was 0.032 (IQR 0.012-0.069), and the mean FQ was 0.049 (±0.062). These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 [10][11][12][13][14]17,18,21,23,[29][30][31]36,38,42,[47][48][49][50][51][52][53][55][56][57] and FQ of 0.048. [10][11][12]18,21,23,31,[47][48][49][50][51]53 The hypothesis of this study was confirmed, with more than half (78%) of outcomes analyzed having a loss to follow-up greater than the fragility index for that outcome.…”

“…These values are consistent with the current orthopaedic literature reporting an average median FI of 3.81 [10][11][12][13][14]17,18,21,23,[29][30][31]36,38,42,[47][48][49][50][51][52][53][55][56][57] and FQ of 0.048. [10][11][12]18,21,23,31,[47][48][49][50][51]53 The hypothesis of this study was confirmed, with more than half (78%) of outcomes analyzed having a loss to follow-up greater than the fragility index for that outcome. Furthermore, outcomes from studies with a greater risk of bias and rerupture as an individual outcome were more fragile than the other outcomes observed in this analysis.…”

Fragility Part I: a guide to understanding statistical power

“…The term fragility has recently gained traction in the clinical research community, and in particular, within the field of orthopaedic research [6,[13][14][15]. However, before defining fragility, a topic to be discussed later in this series, a more traditional and perhaps familiar term, i.e., (statistical) power is defined.…”