Analysis of 567,758 randomized controlled trials published over 30 years reveals trends in phrases used to discuss results that do not reach statistical significance

Abstract

The power of language to modify the reader’s perception of interpreting biomedical results cannot be underestimated. Misreporting and misinterpretation are pressing problems in randomized controlled trials (RCT) output. This may be partially related to the statistical significance paradigm used in clinical trials centered around a P value below 0.05 cutoff. Strict use of this P value may lead to strategies of clinical researchers to describe their clinical results with P values approaching but not reaching the threshold to be “almost significant.” The question is how phrases expressing nonsignificant results have been reported in RCTs over the past 30 years. To this end, we conducted a quantitative analysis of English full texts containing 567,758 RCTs recorded in PubMed between 1990 and 2020 (81.5% of all published RCTs in PubMed). We determined the exact presence of 505 predefined phrases denoting results that approach but do not cross the line of formal statistical significance (P < 0.05). We modeled temporal trends in phrase data with Bayesian linear regression. Evidence for temporal change was obtained through Bayes factor (BF) analysis. In a randomly sampled subset, the associated P values were manually extracted. We identified 61,741 phrases in 49,134 RCTs indicating almost significant results (8.65%; 95% confidence interval (CI): 8.58% to 8.73%). The overall prevalence of these phrases remained stable over time, with the most prevalent phrases being “marginally significant” (in 7,735 RCTs), “all but significant” (7,015), “a nonsignificant trend” (3,442), “failed to reach statistical significance” (2,578), and “a strong trend” (1,700). The strongest evidence for an increased temporal prevalence was found for “a numerical trend,” “a positive trend,” “an increasing trend,” and “nominally significant.” In contrast, the phrases “all but significant,” “approaches statistical significance,” “did not quite reach statistical significance,” “difference was apparent,” “failed to reach statistical significance,” and “not quite significant” decreased over time. In a random sampled subset of 29,000 phrases, the manually identified and corresponding 11,926 P values, 68,1% ranged between 0.05 and 0.15 (CI: 67. to 69.0; median 0.06). Our results show that RCT reports regularly contain specific phrases describing marginally nonsignificant results to report P values close to but above the dominant 0.05 cutoff. The fact that the prevalence of the phrases remained stable over time indicates that this practice of broadly interpreting P values close to a predefined threshold remains prevalent. To enhance responsible and transparent interpretation of RCT results, researchers, clinicians, reviewers, and editors may reduce the focus on formal statistical significance thresholds and stimulate reporting of P values with corresponding effect sizes and CIs and focus on the clinical relevance of the statistical difference found in RCTs.

Spin in the Abstracts of Meta-analyses and Systematic Reviews: Quadriceps Tendon Graft for Anterior Cruciate Ligament Reconstruction

“…Additionally, Boutron et al observed that clinicians were more interested in reading the full text of articles with spin in their abstracts as compared with abstracts with no spin (P = .029). As Chiu et al pointed out, not only does spin affect clinical decision-making, but it also 22 examined .500,000 RCTs and found that .49,000 contained wording to try to express a statistical relationship even when one did not exist, a variant of type 3 spin.…”

How stra(i)nge are your controls? A comparative analysis of metabolic phenotypes in commonly used C57 substrains

“…Data are presented as scatter plots with means ± SD. For all analyses, alpha was set to 5% and p < 0.09 considered as 'marginally significant' as per recent recommendations [25]. It is acknowledged that in the female cohort, low n's maybe a contributing factor for the observance of marginal significances rather than outright significant differences.…”

Not surgical technique, but etiology, contralateral MRI, prior surgery, and side of surgery determine seizure outcome after pediatric hemispherotomy

“…As classical frequentist models are known for their inability to provide evidence in favor of the null hypothesis, we compared the logistic models of seizure outcome and major complications with and without the inclusion of surgery type by the Bayes factor (BF) 30 to assess the role of surgery type in explaining the seizure outcome. 31 Statistical significance was established at two-sided p-values of ≤.05. All statistical analyses were performed by RStudio (2022.02.3).…”