The fragility of statistically significant findings from randomized trials in head and neck surgery

Noel, Christopher W.; McMullen, Caitlin; Yao, Christopher M. K. L.; Monteiro, Eric; Goldstein, David P.; Eskander, Antoine; Almeida, John R. de

doi:10.1002/lary.27183

Cited by 27 publications

(17 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…82 RCTs had the FI less than the number of missing data points. Noel 17 Head and neck surgery 27 RCTs. The median FI was 8 (IQR 2.25-18.25).…”

Section: Rct With 1-1 Randomization and Continuous Outcomesmentioning

confidence: 99%

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

2021

J GEN INTERN MED

View full text Add to dashboard Cite

I n a randomized controlled trial (RCT) comparing treatments A and B, a null hypothesis (H 0 ) of no difference in a primary outcome of interest is defined. Whether any observed difference is statistically significant has traditionally been based on the P value and the confidence interval (CI). For a century, an arbitrary threshold of 0.05 (1/20) has been used to define statistical significance. 1 Because this probability is quite low, we conclude that P ≤ 0.05 suggests that the observed difference between A and B is incompatible with H 0 , and, with a ≥ 95% degree of certainty, that A and B are different. The 95% CI is a lower-upper limit in which the true effect estimate lies within a 95% certainty. A 95% CI that does not include the null effect size indicates the observed difference has reached statistical significance, and H 0 is rejected. The 95% CI provides information about the precision of the estimate and complements the P value. Reporting both is recommended.Use of a P value of 0.05 to dichotomize whether treatments A and B are "truly" different is appealing because of its simplicity. Indeed, for all its many limitations, 2 its use actually increased from 1990 to 2015 in MEDLINE and PubMed Central abstracts and articles. 3 Statistical significance carries considerable weight. Researchers, editors and reviewers, readers, and the press tend to become more excited about positive results. 4 However, a shocking number of scientific studies and meta-analyses are not reproducible or replicable. [5][6][7] A reasonable question to ask is: if a positive study was to be repeated, how "easy" might it be for the results to change from being statistically significant to non-significant (and thus, rightly or wrongly, lose some of their appeal)? What if there was, by sheer chance, one or several more (or less) outcome event(s) in one of the comparative groups? Would the coveted statistical significance be lost? In recent years, there has been much interest in a metric-the fragility index (FI), [8][9][10][11] first proposed 3 decades ago 12 -to test the robustness/fragility of statistically significant results. The various applications and implications of the FI are discussed herein.

show abstract

“…82 RCTs had the FI less than the number of missing data points. Noel 17 Head and neck surgery 27 RCTs. The median FI was 8 (IQR 2.25-18.25).…”

Section: Rct With 1-1 Randomization and Continuous Outcomesmentioning

confidence: 99%

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

2021

J GEN INTERN MED

View full text Add to dashboard Cite

show abstract

“…The FI was initially proposed by Feinstein in 1990 and has helped inform a body of literature that emphasizes the statistical fragility of findings across various medical disciplines. 15 16 17 18 19 20 21 22 It has been applied across multiple orthopaedic subspecialties including spine, sport, trauma, and shoulder surgery. 18 , 23 - 25 To further account for the differences in sample size, Ahmed et al 26 proposed the concept of a fragility quotient (FQ), which is a measure of quantitative significance and is determined by dividing the FI by the sample size.…”

mentioning

confidence: 99%

The Fragility of Statistical Findings in Achilles Tendon Injury Research: A Systematic Review

et al. 2021

View full text Add to dashboard Cite

Introduction: Probability analysis with the reporting of P values is often used to determine the statistical significance of study findings in the Achilles literature. The purpose of this study was to determine the utility of applying a fragility analysis to comparative trials evaluating Achilles tendon injuries. Methods: We identified all dichotomous outcome data for comparative studies of Achilles tendon injuries published in 11 orthopaedic journals from 2000 to 2020. The fragility index (FI) was determined by the number of event reversals required to change a P value from less than 0.05 to greater than 0.05, or vice-versa. The associated fragility quotient was determined by dividing the FI by the sample size. Results: Of the 51,357 studies screened, 1,487 met the search criteria, with 51 comparative studies and 177 total outcome events included for analysis. The overall FI was only 4 with an associated fragility quotient of 0.048. One-half of the studies failed to report lost to follow-up data, with an additional 21.6% reporting loss to follow-up of greater than or equal to 4. Conclusion: Our fragility analysis suggests that Achilles tendon injury outcomes are not as statistically stable as previously thought and should be interpreted with caution.

show abstract

“…Despite this, a number of formal systematic review methods were utilized, including the use of multiple reviewers and meta-analytical techniques. Reviews are inherently limited by the evidence quality available in the literature, and as randomized controlled trials were both few in number and high in risk of bias limitations in conclusions drawn from our findings must be noted [82]. However, this study does offer the most comprehensive review of the contemporary literature on the non-surgical management of oral cavity cancer.…”

Section: Limitationsmentioning

confidence: 91%

Nonsurgical management of resectable oral cavity cancer in the wake of COVID-19: A rapid review and meta-analysis

Forner

Noel

et al. 2020

Oral Oncology

Self Cite

View full text Add to dashboard Cite

Surgery is the preferred treatment modality for oral squamous cell carcinoma (OSCC). However, due to limited resources, re-assessment of treatment paradigms in the wake of the Coronavirus Disease 2019 (COVID-19) pandemic is urgently required. In this rapid review, we described contemporary oncological outcomes for OSCC using non-surgical modalities. Methods: A systematic literature search was conducted for articles published between January 1, 2010 and April 1, 2020 on MEDLINE and Cochrane CENTRAL. Studies were included if they contained patients with OSCC treated with either neoadjuvant, induction, or definitive radiotherapy, chemotherapy, immunotherapy, or combination thereof, and an outcome of overall survival. Results: In total, 36 articles were included. Definitive radiotherapy or chemoradiotherapy were the focus of 18 articles and neoadjuvant chemotherapy or chemoradiotherapy were the focus of the other 18 articles. In early stage OSCC, definitive radiotherapy, with or without concurrent chemotherapy, was associated with a significantly increased hazard of death compared to definitive surgery (HR: 2.39, 95% CI: 1.56-3.67, I 2 : 63%). The hazard of death was non-significantly increased with definitive chemoradiotherapy in studies excluding early disease (HR: 1.98, 95% CI: 0.85-4.64, I 2 : 84%). Two recent randomized control trials have been conducted, demonstrating no survival advantage to neoadjuvant chemotherapy. Conclusion: This review suggests that primary radiotherapy and chemoradiotherapy are inferior to surgical management for OSCC. Strategies for surgical delay warranting consideration are sparse, but may include several neoadjuvant regimens, recognizing these regimens may not offer a survival benefit over definitive surgery alone.

show abstract

The fragility of statistically significant findings from randomized trials in head and neck surgery

Abstract: Randomized trials in the head and neck cancer literature where surgery is a primary modality are relatively nonrobust statistically with low FI scores. Laryngoscope, 128:2094-2100, 2018.

Cited by 27 publications

References 55 publications

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

The Fragility Index for Assessing the Robustness of the Statistically Significant Results of Experimental Clinical Studies

The Fragility of Statistical Findings in Achilles Tendon Injury Research: A Systematic Review

Nonsurgical management of resectable oral cavity cancer in the wake of COVID-19: A rapid review and meta-analysis

Contact Info

Product

Resources

About