The statistical significance filter leads to overoptimistic expectations of replicability

Abstract

We show that publishing results using the statistical significance filter-publishing only when the p-value is less than 0.05-leads to a vicious cycle of overoptimistic expectation of the replicability of results. First, we show analytically that when true statistical power is relatively low, computing power based on statistically significant results will lead to overestimates of power. Then, we present a case study using 10 experimental comparisons drawn from a recently published metaanalysis in psycholinguistics . We show that the statistically significant results yield an illusion of replicability. This illusion holds even if the researcher doesn't conduct any formal power analysis but just uses statistical significance to informally assess robustness (i.e., replicability) of results.

The Effect of Prominence and Cue Association on Retrieval Processes: A Computational Account

“…As discussed in J€ ager et al (2017) and Vasishth et al (2018), low power and publication bias could be important factors that weaken the empirical claims. As discussed in J€ ager et al (2017) and Vasishth et al (2018), low power and publication bias could be important factors that weaken the empirical claims.…”

“…Some caution is also needed as regards the interpretation of the available data. As discussed in J€ ager et al (2017) and Vasishth et al (2018), low power and publication bias could be important factors that weaken the empirical claims. Appendix B in J€ ager et al (2017) shows that power for many of the published studies on interference could be as low as 10%-20%.…”

“…For example, how do LV05's predictions fare in target-mismatch reflexives/reciprocals? In Vasishth et al (2018), we provide further discussions of this general point about the adverse consequences of low power on developing an empirical base for theory testing, and provide constructive suggestions on how the situation could be improved. Dillon et al (2013) argued that the parser was immune to facilitatory interference based on a 40-subject study.…”

“…This example illustrates the need for obtaining more precise estimates of the effects of interest than we currently have. In Vasishth et al (2018), we provide further discussions of this general point about the adverse consequences of low power on developing an empirical base for theory testing, and provide constructive suggestions on how the situation could be improved. The data on non-agreement subject-verb dependencies agree overall with the general LV05 predictions-inhibition in target-match configurations-and thus had a good fit in both models.…”

The World of Research Has Gone Berserk: Modeling the Consequences of Requiring “Greater Statistical Stringency” for Scientific Publication

“…Hence a study which really attains 80% power is not likely to suffer in its quest for publication, motivating c 95 = 0.8. However, it is well commented upon (Bland 2009, Vasishth & Gelman 2017) that often a-priori sample size claims are exaggerated through various mechanisms, meaning that a study with less than 80% power might be advertised as having 80% power. This is the basis for setting c 50 = 0.5, i.e., truly possessing only 50% power does substantially reduce, but not eliminate, the chance of publication.…”