“Statistical Significance” and Statistical Reporting: Moving Beyond Binary

Abstract

Null hypothesis significance testing (NHST) is the default approach to statistical analysis and reporting in marketing and the biomedical and social sciences more broadly. Despite its default role, NHST has long been criticized by both statisticians and applied researchers including those within marketing. Therefore, the authors propose a major transition in statistical analysis and reporting. Specifically, they propose moving beyond binary: abandoning NHST as the default approach to statistical analysis and reporting. To facilitate this, they briefly review some of the principal problems associated with NHST. They next discuss some principles that they believe should underlie statistical analysis and reporting. They then use these principles to motivate some guidelines for statistical analysis and reporting. They next provide some examples that illustrate statistical analysis and reporting that adheres to their principles and guidelines. They conclude with a brief discussion.

A framework for public health analyses: a list of user-friendly solutions to avoid significance misconceptions in statistical testing

“…Three years later, 800 scientists signed a petition to propose the definitive abandonment of the originally problematic and now completely distorted concept of statistical significance . Such efforts continue to this day (McShane et al, 2023;. Nevertheless, the following errors remain extremely common (Greenland et al, 2016;Amrhein & Greenland, 2018): i) Dichotomania: The results are divided into (statistically) significant and non-significant based on an arbitrary threshold.…”

“…Nevertheless, with due reservations, it is still possible to largely adhere to this fashion while ensuring interpretative correctness. Specifically, the P-value can be adopted as a continuous measure of the incompatibility (significance) of the data with the statistical t-assumption in the best-case scenario (McShane et al, 2023). Possible ranges of statistical incompatibility (significance) are shown in Table 3c, attempting to remain consistent with Tables 3a and 3b.…”

When significant is not significant: Six clinical examples that disprove common wrong beliefs about statistical testing

“…Since the original formulation by Sir Ronald Fisher in the early 1920s, the concept of statistical significance has been subject to serious misinterpretations. Despite more than 100 years having passed, these criticalities remain as vivid today as they were back then, if not more so (Wasserstein & Lazar, 2016;Gelman, 2018;Amrhein et al, 2019 a ;Greenland et al, 2022;McShane et al, 2023;Mansournia & Nazemipour, 2024). Given that the misuse of statistical testing in public health can lead to highly dangerous outcomes such as the approval of ineffective treatments or the rejection of effective ones, in this brief letter, we present a series of examples aimed at definitively dispelling some of the most common and erroneous beliefs about statistical significance.…”