Manipulating the alpha level cannot cure significance testing – comments on "Redefine statistical significance"

Abstract

We argue that depending on p-values to reject null hypotheses, including a recent call for changing the canonical alpha level for statistical significance from .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable criterion levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and determining sample sizes much more directly than significance testing does; but none of the statistical tools should replace significance testing as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, or implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

Quantifying the prevalence of assortative mating in a human population

“…Statistics is often more misleading than useful (17,18). Comparing the randomly generated results with the empirically obtained frequency distribution produces probability values of rejecting the null hypothesis close to absolute 0.…”

Retract p < 0.005 and propose using JASP, instead

“…“I find it especially troubling”—she continues—“to spoze an error statistician…ought to use a Bayes Factor as the future gold standard for measuring his error statistical tool…even though Bayes Factors don’t control or measure error probabilities” (2017b). Furthermore, Mayo pinpoints the (old) fallacy of transposing the conditional, whereby the (error) probability of a test is confused with the (posterior) probability of a belief (also Trafimow et al , 2017 ). And despite “60 years (sic) old…demonstrations [showing] that with reasonable tests and reasonable prior probabilities, the disparity vanishes…they still mean different things” (2017c).…”

Retract p < 0.005 and propose using JASP, instead

“…Lakens et al (2017) add lack of experimental redundancy, logical traps, research opacity, and poor accounting of sources of error, as well as the risks of reduced generalisability and research breadth were Benjamin et al ’s proposal to succeed. Methodological concerns were also raised by Amrhein & Greenland (2017); Black (2017); Byrd (2017); Chapman (2017); Crane (2017); Ferreira & Henderson (2017); Greenland (2017); Hamlin (2017); Kong (2017); Lew (2017); Llewelyn (2017); Martin (2017); McShane et al (2017); Passin (2017); Steltenpohl (2017); Trafimow et al (2017); Young (2017); Zollman (2017); and Morey (2017). Some researchers even propose the use of preregistration as a way of minimizing above problems ( Hamlin, 2017; Llewelyn, 2017; van der Zee, 2017)…”

“…Addressing replication directly, Chapman (2017) and Trafimow et al (2017) point out that the problem with replication is not too many false positives but insufficient power. Krueger (2017; also McShane et al , 2017, Trafimow et al , 2017) chides Benjamin et al for the incoherence of considering replication as order-dependent and inverting the exploratory-confirmatory nature of replication by proposing to make the former more difficult to achieve and the latter more liberal.…”

“…Krueger (2017; also McShane et al , 2017, Trafimow et al , 2017) chides Benjamin et al for the incoherence of considering replication as order-dependent and inverting the exploratory-confirmatory nature of replication by proposing to make the former more difficult to achieve and the latter more liberal. He further chides them on whether they would disallow their own past findings at the 5% threshold.…”