Manipulating the alpha level cannot cure significance testing

Abstract

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p= .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

False confidence, non-additive beliefs, and valid statistical inference

“…Still, it makes sense to ask if contributions from the statistics community have been helpful. The dust is perhaps still settling, but, based on the expert commentary supplementing the ASA statement, the follow-up by Ionides et al (2017), the recent proposal of Benjamin et al (2017) to simply change the default "p < 0.05" cutoff to "p < 0.005," and the plethora of criticism that followed (e.g., Crane 2018b; Lakens et al 2018;McShane et al 2018;Trafimow et al 2018), it seems that statisticians' views are as diverse as ever.…”

Twenty steps towards an adequate inferential interpretation of p-values

“…The American Economic Review and Econometrica furthermore request authors to explicitly report effect sizes and display standard errors (or even confidence intervals) instead of p-values in results tables, but they do not explicitly ban p-values. This is consistent with a suggestion put forward by many critical voices in the recent debateto demote p-values from their pedestal and consider them as a tool amongst many that help make appropriate inferences (cf., e.g., Amrhein et al 2017;McShane et al 2017;Trafimow et al 2018). The fact that some leading economics journals, which are widely considered as beacons for best practice, have initiated modest but sensible changes with regard to the use of the p-value is a promising signal.…”

“…Dichotomization in conjunction with misleading terminology propagate cognitive biases that seduce researchers to make logically inconsistent and overconfident inferences, both when p is below and when it is above the "significance" threshold. The following errors seem to be particularly widespread: 1 1) use of p-values when there is neither random sampling nor randomization 2) confusion of statistical and practical significance or complete neglect of effect size 3) unwarranted binary statements of there being an effect as opposed to no effect, coming along with -misinterpretations of p-values below 0.05 as posterior probabilities of the null hypothesis -mixing up of estimating and testing and misinterpretation of "significant" results as evidence confirming the coefficients/effect sizes estimated from a single sample treatment of "statistically non-significant" effects as being zero (confirmation of the null) 4) inflation of evidence caused by unconsidered multiple comparisons and p-hacking 5) inflation of effect sizes caused by considering "significant" results only 1 See, for example, McCloskey and Ziliak (1996), Sellke et al (2001), Ioannidis (2005), Ziliak and McCloskey (2008), Krämer (2011), Ioannidis and Doucouliagos (2013), Kline (2013), Colquhoun (2014), Gelman and Loken (2014), Motulsky (2014), Vogt et al (2014), Gigerenzer and Marewski (2015), Greenland et al (2016), Hirschauer et al (2016;2018), Wasserstein and Lazar (2016), Ziliak (2016), Amrhein et al (2017), and Trafimow et al (2018). This list contains but a small selection of the literature on p-value misconceptions from the last 20 years.…”

“…While Berry (2017: 896) might be pushing too hard when claiming that a pvalue "as such has no inferential content," one must recognize that it is but a graded measure of the strength of evidence against the null, but only in the sense that small p-values will occur more often if there is an effect compared to no effect (Hirschauer et al 2018). 2 Joining Berry (2017), Gelman and Carlin (2017), Greenland (2017), McShane and Gal (2017), McShane et al (2017), Trafimow et al (2018), and many others, we believe that degrading the p-value's continuous message into binary "significance" declarations ("bright line rules") is at the heart of the problem. 3 Since the p-value is deeply anchored in the minds of most scientists including economists, we believe that demanding drastic changes, such as renouncing p-values or replacing frequentist approaches by Bayesian methods, is not the most promising way to guard researchers from the inferential errors that we see today.…”

“…Quite on the contrary. "Under reasonable sample sizes and reasonable population effect sizes, it is the abnormally large sample effect sizes that result in p-values that meet the 0.05 level, or the 0.005 level, or any other alpha level, as is obvious from the standpoint of statistical regression" (Trafimow et al 2018). Hence, even seemingly neutral representations such as "the retail prices of product A exceed the retail prices of product B by 20% on average (p < 0.001)" may be misleading because they insinuate that the evidence against the null can be translated into evidence in favor of the concrete effect that we happened to find in a single sample (Amrhein et al 2017).…”