P-values – a chronic conundrum

Abstract

Background In medical research and practice, the p-value is arguably the most often used statistic and yet it is widely misconstrued as the probability of the type I error, which comes with serious consequences. This misunderstanding can greatly affect the reproducibility in research, treatment selection in medical practice, and model specification in empirical analyses. By using plain language and concrete examples, this paper is intended to elucidate the p-value confusion from its root, to explicate the difference between significance and hypothesis testing, to illuminate the consequences of the confusion, and to present a viable alternative to the conventional p-value. Main text The confusion with p-values has plagued the research community and medical practitioners for decades. However, efforts to clarify it have been largely futile, in part, because intuitive yet mathematically rigorous educational materials are scarce. Additionally, the lack of a practical alternative to the p-value for guarding against randomness also plays a role. The p-value confusion is rooted in the misconception of significance and hypothesis testing. Most, including many statisticians, are unaware that p-values and significance testing formed by Fisher are incomparable to the hypothesis testing paradigm created by Neyman and Pearson. And most otherwise great statistics textbooks tend to cobble the two paradigms together and make no effort to elucidate the subtle but fundamental differences between them. The p-value is a practical tool gauging the “strength of evidence” against the null hypothesis. It informs investigators that a p-value of 0.001, for example, is stronger than 0.05. However, p-values produced in significance testing are not the probabilities of type I errors as commonly misconceived. For a p-value of 0.05, the chance a treatment does not work is not 5%; rather, it is at least 28.9%. Conclusions A long-overdue effort to understand p-values correctly is much needed. However, in medical research and practice, just banning significance testing and accepting uncertainty are not enough. Researchers, clinicians, and patients alike need to know the probability a treatment will or will not work. Thus, the calibrated p-values (the probability that a treatment does not work) should be reported in research papers.

In-Vitro Inactivation of Sabin-Polioviruses for Development of Safe and Effective Polio Vaccine

“…Another explanation could be the use of “statistical significance” itself in the interpretation of results. In the last few years, many calls for omitting the use of P value and statistical significance to accept or reject the research hypothesis were made [ 43 , 44 ]. The statistically non- significant differences do not mean that the data are not different; they rather indicate that many factors such as the sample size and the number of replicates can affect the experiment and consequently might give different results [ 45 ].…”

COVID-19 diagnosis from routine blood tests using artificial intelligence techniques

“…The value of the SHAP determines the importance of each feature, the high or low value of each feature, and its impact based on the model output [29] . To interpret the final model of this study, we used tools such as SHAP, ELI5, and LIME [29] , [30] , [31] . This way, we can understand the internal performance of the model and gain insight into the application.…”

The Role of GI Products or Local Products in the Environment—Consumer Awareness and Preferences in Albania, Bulgaria and Poland

“…In addition, we used various statistical tests such as Chi-square (χ 2 ), p-value to compare the "observed significance level" for the test hypothesis [161]. The p-value is a convenient tool measuring the "strength of evidence" against the null hypothesis [162]. The largest share of respondents, both in Albania and Poland, was in the 21-30 age group.…”