Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations

Abstract

Misinterpretation and abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet remain rampant. A key problem is that there are no interpretations of these concepts that are at once simple, intuitive, correct, and foolproof. Instead, correct use and interpretation of these statistics requires an attention to detail which seems to tax the patience of working scientists. This high cognitive demand has led to an epidemic of shortcut definitions and interpretations that are simply wrong, sometimes disastrously so—and yet these misinterpretations dominate much of the scientific literature. In light of this problem, we provide definitions and a discussion of basic statistics that are more general and critical than typically found in traditional introductory expositions. Our goal is to provide a resource for instructors, researchers, and consumers of statistics whose knowledge of statistical theory and technique may be limited but who wish to avoid and spot misinterpretations. We emphasize how violation of often unstated analysis protocols (such as selecting analyses for presentation based on the P values they produce) can lead to small P values even if the declared test hypothesis is correct, and can lead to large P values even if that hypothesis is incorrect. We then provide an explanatory list of 25 misinterpretations of P values, confidence intervals, and power. We conclude with guidelines for improving statistical interpretation and reporting.

Manipulating the alpha level cannot cure significance testing

“…In addition to the obvious costs of obtaining large sample sizes, there may be an underappreciation of how much sample size matters (Vankov et al, 2014), of the importance of incentives to favor novelty over replicability (Nosek et al, 2012) and of a prevalent misconception that the complement of p-values measures replicability (Cohen, 1994;Thompson, 1996;Greenland et al, 2016). A focus on sample size suggests an alternative to significance testing.…”

“…This goal seems to demand-if one is a Bayesian-that the posterior probability of the null hypothesis should be low given the obtained finding. But the p-value one obtains is the probability of the finding, and of more extreme findings, given that the null hypothesis and all other assumptions about the model were correct (Greenland et al, 2016;Greenland, 2017), and one would need to make an invalid inverse inference to draw a conclusion about the probability of the null hypothesis given the finding.…”

Patient and Hospital Characteristics Associated with Interhospital Transfer for Adults with Ventilator-Dependent Respiratory Failure

“…Descriptive statistics of both the transferred and nontransferred cohorts were calculated; however, hypothesis testing of bivariate statistical differences between the transferred and nontransferred groups are not displayed, because the large sample size resulted in statistically significant but clinically irrelevant differences in nearly all variables (26,27). Manually fitted multivariable logistic regression was used to identify the associations between patient variables and hospital category and the odds of an interhospital transfer.…”

“…To assess if variation differences caused by the nesting of patients within hospitals significantly impacted model performance, a final parsimonious multilevel generalized linear mixed model, also known as a hierarchical model, was fitted using the SAS/STAT 9.1 production GLIMMIX procedure, which allows for the clustering of patients within hospitals (30). The same purposeful selection criteria as described above were used for the multilevel model (27). Model performance was evaluated by c-statistics.…”

The limitations of opportunistic epidemiology, pseudopod epidemiology

“…Epidemiology studies were loaded with numerous ''p values'' (the peppered p value syndrome). Whether these variables were in the causal pathway or the possible pathophysiological basis of the statistically significant associations were not necessarily a high priority [23,24].…”