Seeking to address the lack of research reproducibility in science, including psychology and the life sciences, a pragmatic solution has been raised recently: to use a stricter < 0.005 standard for statistical significance when p claiming evidence of new discoveries. Notwithstanding its potential impact, the proposal has motivated a large mass of authors to dispute it from different philosophical and methodological angles. This article reflects on the original argument and the consequent counterarguments, and concludes with a simpler and better-suited alternative that the authors of the proposal knew about and, perhaps, should have made from their Jeffresian perspective: to use a Bayes factors analysis in parallel (e.g., via JASP) in order to learn more about frequentist error statistics and about Bayesian prior and posterior beliefs without having to mix inconsistent research philosophies.
Keywords
ArgumentSeeking to address the lack of research reproducibility due to the high rate of false positives in the literature, Benjamin et al. (2017a); Benjamin et al. (2017b) propose a pragmatic solution which "aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance" (also Savehn, 2017): to use a stricter p < 0.005 standard for statistical significance when claiming evidence of new discoveries.The proposal is subject to several constrains in its application: (1) to claims of discovery of new effects (thus, not necessarily to replication studies); (2) when using null hypothesis significance testing (arguably Fisher's approach, perhaps even Neyman-Pearson's, but excluding other p-value-generating approaches such as resampling); (3) in fields with too flexible standards (namely 5% or above); (4) when the prior odds of alternative-to-null hypothesis is in the range 1-to-5, to 1-to-40 (stricter standards are required with lower odds); (5) for researcher's consumption (thus, not a standard for journal rejection, although "journals can help transition to the new statistical significance threshold"; also, "journals editors and funding institutions could easily enforce the proposal", Wagenmakers, 2017; and "its implementation only requires journal editors to agree on the new threshold", Machery, 2017); (6) while still keeping findings with probability up to 5% as suggestive (and meriting publication if so "properly labelled"); (7) despite many of the proponents believing that the proposal is nonsense, anyway (that is, it is a quick fix, not a credible one; also Ioannidis in Easwaran, 2017; Resnick, 2017; Wagenmakers, 2017; Wagenmakers & Gronau, 2017).
Amendments from Version 1Minor changes incorporating reviewers' recommendations:The legend in Figure 1 now defines the acronyms in the figure.• [2] A new reference to Perezgonzalez (2015) now implies that the pseudoscientific label attached to the NHST element ( Figure 1) follows from the rhetoric in such reference.• [3] A second note clarifies that JASP also allows to use Cauchy, Normal and t-distributions as informed prio...