Beyond Power Calculations

Gelman, Andrew; Carlin, John B.

doi:10.1177/1745691614551642

Cited by 1,008 publications

(536 citation statements)

References 32 publications

Supporting

Mentioning

465

Contrasting

Unclassified

Order By: Relevance

“…Hence, considerable resources would be saved by not performing future studies based on false premises. Increasing sample sizes is also desirable because studies with small sample sizes tend to yield inflated effect size estimates 11 , and publication and other biases may be more likely in an environment of small studies 12 . We believe that efficiency gains would far outweigh losses.…”

Section: Potential Objectionsmentioning

confidence: 99%

Redefine statistical significance

et al. 2017

View full text Add to dashboard Cite

We propose to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries. T he lack of reproducibility of scientific studies has caused growing concern over the credibility of claims of new discoveries based on 'statistically significant' findings. There has been much progress toward documenting and addressing several causes of this lack of reproducibility (for example, multiple testing, P-hacking, publication bias and under-powered studies). However, we believe that a leading cause of non-reproducibility has not yet been adequately addressed: statistical standards of evidence for claiming new discoveries in many fields of science are simply too low. Associating statistically significant findings with P < 0.05 results in a high rate of false positives even in the absence of other experimental, procedural and reporting problems.For fields where the threshold for defining statistical significance for new discoveries is P < 0.05, we propose a change to P < 0.005. This simple step would immediately improve the reproducibility of scientific research in many fields. Results that would currently be called significant but do not meet the new threshold should instead be called suggestive. While statisticians have known the relative weakness of using P ≈ 0.05 as a threshold for discovery and the proposal to lower it to 0.005 is not new 1,2 , a critical mass of researchers now endorse this change.We restrict our recommendation to claims of discovery of new effects. We do not address the appropriate threshold for confirmatory or contradictory replications of existing claims. We also do not advocate changes to discovery thresholds in fields that have already adopted more stringent standards (for example, genomics and high-energy physics research; see the 'Potential objections' section below).We also restrict our recommendation to studies that conduct null hypothesis significance tests. We have diverse views about how best to improve reproducibility, and many of us believe that other ways of summarizing the data, such as Bayes factors or other posterior summaries based on clearly articulated model assumptions, are preferable to P values. However, changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance.

show abstract

Section: Potential Objectionsmentioning

confidence: 99%

Redefine statistical significance

et al. 2017

View full text Add to dashboard Cite

show abstract

“…This implies that a single replication attempt, with a chance of 23% of incorrectly not detecting an existing effect of the original size, is not enough to conclude that the effect does not exist (at least when one would rely on the 5% significance threshold). The latter is amplified by the conclusion that most effects are overestimations, and hence, true to‐be‐replicated effects are smaller than those that are reported (Gelman & Carlin, 2014). Therefore, we can conclude that we could not replicate the original effect of identical size but we cannot with high confidence ascertain that the effect (i.e., more specific time references in truthful than in deceptive intentions) does not exist.…”

Section: Discussionmentioning

confidence: 84%

“…Treating the original effect size (here d = 0.54) at face value can be misleading because most published effect sizes are overestimations of the true effect (Gelman & Carlin, 2014; Simonsohn, 2015). 2

We thank Timothy Luke for pointing us in to that direction during the reviewing process.

To avoid inflating evidence for the null hypothesis, we also calculated the informed prior Bayes factor estimation using a corrected original effect size of 75% ( d = 0.41, BF 01 = 4.23), 50% ( d = 0.27, BF 01 = 2.41), 25% ( d = 0.14, BF 01 = 1.59), and 10% of the original ( d = 0.05, BF 01 = 1.33).…”

Section: Non‐preregistered Analysesmentioning

confidence: 99%

The first direct replication on using verbal credibility assessment for the detection of deceptive intentions

Kleinberg

Warmelink

Arntz

et al. 2018

Applied Cognitive Psychology

View full text Add to dashboard Cite

Verbal deception detection has gained momentum as a technique to tell truth‐tellers from liars. At the same time, researchers' degrees of freedom make it hard to assess the robustness of effects. Replication research can help evaluate how reproducible an effect is. We present the first replication in verbal deception research whereby ferry passengers were instructed to tell the truth or lie about their travel plans. The original study found truth‐tellers to include more specific time references in their answers. The replication study that closely mimicked the setting, procedure, materials, coding, and analyses found no lie–truth difference for specific time references. Although the power of our replication study was suboptimal (0.77), Bayesian statistics showed evidence in favor of the null hypothesis. Given the great applied consequences of verbal credibility tests, we hope this first replication attempt ignites much needed preregistered, high‐powered, multilab replication efforts.

show abstract

“…Low statistical power diminishes the probability that experimental findings are true and increases the probability that effect sizes are overestimated (Type M error) or point in the wrong direction (Type S error; Button et al 2013;Gelman & Carlin, 2014;Ioannidis, 2005).…”

Section: What Are the Problems Plaguing Scientific Methodology?mentioning

confidence: 99%

Registered Reports for Student Research

2016

Journal of European Psychology Students

View full text Add to dashboard Cite

The pre-registration of research via registered reports is a recent development in the field of psychology. The aim of pre-registration is to encourage research that presents sound hypotheses and methodology (Chambers, 2014) in order to counter undesirable but prevalent research practices such as cherry-picking and p-hacking. In this Letter from the Editors, we wish to echo calls for registered reports and outline how we, the Editors at the Journal of European Psychology Students (JEPS), plan to introduce registered reports for student research. We address the issues necessitating the introduction of registered reports and outline the approach needed for implementing this initiative in a student journal.

show abstract

Beyond Power Calculations

Cited by 1,008 publications

References 32 publications

Redefine statistical significance

Redefine statistical significance

The first direct replication on using verbal credibility assessment for the detection of deceptive intentions

Registered Reports for Student Research

Contact Info

Product

Resources

About