Manipulating the Alpha Level Cannot Cure Significance Testing

Abstract

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p = 0.05 to p = 0.005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of 0.05, 0.01, 0.005, or anything else, is not acceptable.

Statistical significance: p value, 0.05 threshold, and applications to radiomics—reasons for a conservative approach

“…These arguments are not new. An article signed by 54 authors [18] has provided a similar view, with a deeper technical explanation on why the statements by Ioannidis [15] and Benjamin et al [4] are unjustified. Another article, signed by 88 authors [17], has questioned the idea that the significance threshold should be based on the amount of relative evidence indicated by Bayesian factors 1 , as done by Benjamin et al [4], whose assumptions were considered to be unjustified.…”

Severity Testing: A Primer

“…Thirdly, as indicated earlier, the CI approach allows us to bypass the need for a null hypothesis and, therefore, the whole conundrum of statistical significance (e.g., Lakens et al, 2018;Trafimow et al, 2018;Hurlbert, Levine & Utts, 2019;Amrhein, Greenland & McShane, 2019). For example, let's say we get a score of 151 in some measure with a sample of 100 participants, a measure which we know is about μ = 150 (σ = 10) in the population at large.…”

Presidential address: Six open questions to genetic epidemiologists

“…In this case, we can refer to a number of current recommendations from the literature (Ioannidis, 2018). For example, Benjamin et al (2018) recommended that the usual threshold for statistical significance be generally lowered from 5% to 0.5%, which has been dismissed by others (e.g., Trafimow et al, 2018). In other publications, it was suggested not to use p values anymore but rely on alternative methods instead.…”