Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects

Pereira, Tiago; Ioannidis, John P. A.

doi:10.1016/j.jclinepi.2010.12.012

Cited by 133 publications

(93 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We then calculated the power of each primary study to detect the meta-analytic effect in the corresponding meta-analysis with a ttest for correlation, assuming α = .05 and two-sided tests, using the R package "pwr" (Champely, 2017). Note that the meta-analytic effect size estimates are probably inflated, precisely because of selective reporting of significant findings and researcher degrees of freedom, that inflated the effect estimate (Francis, 2013b;Nuijten, Van Assen, Veldkamp, & Wicherts, 2015;Pereira & Ioannidis, 2011). Furthermore, in random effects meta-analyses small studies receive relatively more weight than in fixed effect meta-analyses.…”

Section: Power In Intelligence Researchmentioning

confidence: 99%

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

show abstract

Section: Power In Intelligence Researchmentioning

confidence: 99%

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Nuijten¹

2018

Preprint

View full text Add to dashboard Cite

show abstract

“…For this sample, we further extracted detailed information on type of outcome, involvement of drug(s) in the comparison (yes/ no), or type of study (randomized or nonrandomized). Outcomes were classified 10 as death, composites including death, clinically defined benefit, laboratory-defined benefit, pain response, withdrawals, and harms. Withdrawals due to specific reasons (eg, lack of clinical benefit or harms) were counted under withdrawals.…”

Section: Categorization Of Types Of Outcomes and Interventionsmentioning

confidence: 99%

“…The overestimation is commensurate with the winner's curse phenomenon (regression-to-the-mean for inflated treatment effects). 6,10,27 Clinical researchers do not seem reluctant to conduct further studies when a prior trial had identified a very large effect. This is not surprising since typically there are many other outcomes on the same comparison.…”

Section: Commentmentioning

confidence: 99%

Empirical Evaluation of Very Large Treatment Effects of Medical Interventions

Pereira¹,

Horwitz²,

Ioannidis³

2012

JAMA

Self Cite

211

164

View full text Add to dashboard Cite

show abstract

“…Several research fields appear to be in crisis of confidence (McNutt, 2014;Nuzzo, 2014Nuzzo, , 2015Horton, 2015;Parker et al, 2016) as evidence emerges that the majority of published research findings cannot be replicated (Ioannidis, 2005;Pereira & Ioannidis, 2011;Prinz, Schlange & Asadullah, 2011;Begley & Ellis, 2012;Open Science Collaboration, 2015). According to a recent survey in Nature (Baker, 2016), 52% of researchers believe that there is 'a significant crisis', 38% see 'a slight crisis', and only 3% see 'no crisis'.…”

Section: Introductionmentioning

confidence: 99%

Detecting and avoiding likely false‐positive findings – a practical guide

2016

View full text Add to dashboard Cite

Recently there has been a growing concern that many published research findings do not hold up in attempts to replicate them. We argue that this problem may originate from a culture of 'you can publish if you found a significant effect'. This culture creates a systematic bias against the null hypothesis which renders meta-analyses questionable and may even lead to a situation where hypotheses become difficult to falsify. In order to pinpoint the sources of error and possible solutions, we review current scientific practices with regard to their effect on the probability of drawing a false-positive conclusion. We explain why the proportion of published false-positive findings is expected to increase with (i) decreasing sample size, (ii) increasing pursuit of novelty, (iii) various forms of multiple testing and researcher flexibility, and (iv) incorrect P-values, especially due to unaccounted pseudoreplication, i.e. the non-independence of data points (clustered data). We provide examples showing how statistical pitfalls and psychological traps lead to conclusions that are biased and unreliable, and we show how these mistakes can be avoided. Ultimately, we hope to contribute to a culture of 'you can publish if your study is rigorous'. To this end, we highlight promising strategies towards making science more objective. Specifically, we enthusiastically encourage scientists to preregister their studies (including a priori hypotheses and complete analysis plans), to blind observers to treatment groups during data collection and analysis, and unconditionally to report all results. Also, we advocate reallocating some efforts away from seeking novelty and discovery and towards replicating important research findings of one's own and of others for the benefit of the scientific community as a whole. We believe these efforts will be aided by a shift in evaluation criteria away from the current system which values metrics of 'impact' almost exclusively and towards a system which explicitly values indices of scientific rigour.

show abstract

Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects

Cited by 133 publications

References 64 publications

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science

Empirical Evaluation of Very Large Treatment Effects of Medical Interventions

Detecting and avoiding likely false‐positive findings – a practical guide

Contact Info

Product

Resources

About