The Potential Effect of Lowering the Threshold of Statistical Significance From P < .05 to P < .005 in Orthopaedic Sports Medicine

Misinterpretation of P Values and Statistical Power Creates a False Sense of Certainty: Statistical Significance, Lack of Significance, and the Uncertainty Challenge

“…As noted previously, near the front of this issue of Arthroscopy, readers will find the original scientific article, "The Potential Effect of Lowering the Threshold of Statistical Significance From P < .05 to P < .005 in Orthopaedic Sports Medicine" by Evans et al 2 In addition, near the end of this issue, readers will find the Level V evidence (expert opinion) article, "The Blight of the Type II Error: When No Difference Does Not Mean No Difference" by Domb and Sabetian of the American Hip Institute in Chicago. 19 As mentioned, Evans et al focus on avoidance of falsely positive, albeit statistically significant, conclusions.…”

“…19 As mentioned, Evans et al focus on avoidance of falsely positive, albeit statistically significant, conclusions. 2 In contrast, Domb and Sabetian focus on reasons why studies could fail, errantly, to achieve statistically significant results. Domb and Sabetian introduce that "underpowered studies caused by small sample sizes are especially prevalent in the surgical literature," where power is defined as "the capacity of the study to recognize whether there is a difference (between treatment groups), given that such difference exists."…”

“…There's so much to more to consider when it comes to data, and yet we're still misleadingly dichotomizing study results as significant or not. In the end, readers of both Evans et al 2 and Domb and Sabetian 19 could keep in mind that, moving forward, we need to avoid the rigid and dichotomous thought process of difference versus no difference in favor of a process that estimates uncertainty. When it comes to medical research data, uncertainty, on some level, is omnipresent.…”

“…Near the front of this months' issue of Arthroscopy, readers will find the original scientific article, "The Potential Effect of Lowering the Threshold of Statistical Significance From P < .05 to P < .005 in Orthopaedic Sports Medicine" by Evans, Johnson, Anderson, Checketts, Scott, Middlemist, Fishbeck, and Vassar of the Oklahoma State University. 2 Evans et al 2 report that some respected scholars recommend "redefining statistical significance by changing the P value threshold from .05 to .005" to lower the risk of that medical research studies reach "false-positive" conclusions, and their results show that lowering the P value threshold would dramatically reduce the number of statistically significant findings in randomized controlled trials. This is clinically relevant because if the statistical significance threshold were thus changed, evidence-based recommendations used to guide clinical decision-making would be greatly affected.…”

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

“…In a series of studies, Vassar and colleagues examined the impact of changing the threshold on randomized controlled trials (RCTs) published in general medical, orthopaedic trauma, and orthopaedic sports medicine journals [24][25][26]. The results of these studies are summarized in Table 1, along with a study by Thakur and Jha [27] that examined changing the P value threshold on results from 123 RCTs pertaining to chronic rhinosinusitis and a study by Khan et al [28] that focused on 72 RCTS from high impact general medical and cardiology journals.…”