A Proposed Hybrid Effect Size Plus p -Value Criterion: Empirical Evidence Supporting its Use

Abstract

When the editors of Basic and Applied Social Psychology effectively banned the use of null hypothesis significance testing (NHST) from articles published in their journal, it set off a fire-storm of discussions both supporting the decision and defending the utility of NHST in scientific research. At the heart of NHST is the pvalue which is the probability of obtaining an effect equal to or more extreme than the one observed in the sample data, given the null hypothesis and other model assumptions. Although this is conceptually different from the probability of the null hypothesis being true, given the sample, p-values nonetheless can provide evidential information, toward making an inference about a parameter. Applying a 10,000-case simulation described in this article, the authors found that p-values' inferential signals to either reject or not reject a null hypothesis about the mean (α = 0.05) were consistent for almost 70% of the cases with the parameter's true location for the sampled-from population. Success increases if a hybrid decision criterion, minimum effect size plus p-value (MESP), is used. Here, rejecting the null also requires the difference of the observed statistic from the exact null to be meaningfully large or practically significant, in the researcher's judgment and experience. The simulation compares performances of several methods: from p-value and/or effect sizebased, to confidence-interval based, under various conditions of true location of the mean, test power, and comparative sizes of the meaningful distance and population variability. For any inference procedure that outputs a binary indicator, like flagging whether a p-value is significant, the output of one single experiment is not sufficient evidence for a definitive conclusion. Yet, if a tool like MESP generates a relatively reliable signal and is used knowledgeably as part of a research process, it can provide useful information.

The reign of the p -value is over: what alternative analyses could we employ to fill the power vacuum?

“…It is important to emphasize that these manifold approaches are not all mutually exclusive; for example, many would argue that effect size estimates are an essential component of most analyses. Indeed, Goodman et al [57] go so far as to recommend the use of a hybrid for decision making that requires a low pvalue coupled with an effect size above an a priori determined minimum to be relevant/important in order to reject the null hypothesis. p-values can also be presented alongside Bayes factors for each statistical test conducted ('a B for every p').…”

Moving to a World Beyond “ p < 0.05”

“…The BFB, like the FPR, often clarifies that the evidence against the null hypothesis contained in the p-value is not nearly as strong as is believed by many researchers. Goodman, Spruill, and Komaroff (2019) propose a twostage approach to inference, requiring both a small p-value below a pre-specified level and a pre-specified sufficiently large effect size before declaring a result "significant. " They argue that this method has improved performance relative to use of dichotomized p-values alone.…”

“…Why is eliminating the use of p-values as a truth arbiter so hard? "The basic explanation is neither philosophical nor scientific, but sociologic; everyone uses them, " says Goodman (2019). "It's the same reason we can use money.…”

“…Many factors demand consideration, not the least of which is how to avoid turning every drug decision into a court battle. Goodman (2019) cautions that, even as we seek change, "we must respect the reason why the statistical procedures are there in the first place. " Perhaps the ASA could convene a panel of experts, internal and external to FDA, to provide a workable new paradigm.…”

Statistical significance: p value, 0.05 threshold, and applications to radiomics—reasons for a conservative approach

“…In summary, all of the many methods compared in the study by Goodman et al [38] have strengths and weaknesses and none of them generates an automatic final answer for a definitive inference. These methods are not so much different to one another and often provide similar interpretations of uncertainty [39].…”