Measuring the effects of publication bias in political science

Abstract

Prior research finds that statistically significant results are overrepresented in scientific publications. If significant results are consistently favored in the review process, published results could systematically overstate the magnitude of their findings even under ideal conditions. In this paper, we measure the impact of this publication bias on political science using a new data set of published quantitative results. Although any measurement of publication bias depends on the prior distribution of empirical relationships, we determine that published estimates in political science are on average substantially larger than their true value under a variety of reasonable choices for this prior. We also find that many published estimates have a false positive probability substantially greater than the conventional α = 0.05 threshold for statistical significance if the prior probability of a null relationship exceeds 50%. Finally, although the proportion of published false positives would be reduced if significance tests used a smaller α, this change would not solve the problem of upward bias in the magnitude of published results.

Forschung als soziales Dilemma: Eine meta-wissenschaftliche Bestandsaufnahme zur Glaubwürdigkeit politikwissenschaftlicher Befunde und ein Appell zur Veränderung akademischer Anreizstrukturen

“…Here the hypothesis of no publication bias was rejected at a probability of 1 to 32 billion (Gerber 2008). What is more, the likelihood of publication increases when effect sizes are larger (Esarey/Wu 2016). 9 To quantify the extent of this bias, Franco et al (2014) traced the fortunes of a clearly specified and known set of conducted research projects in political science and examined whether a conducted research project was more likely to be published when the study's outcome coincided with the preferences of academic journals.…”

Legal Empowerment and Group-Based Inequality

“…More critically, it is worth noting inter alia that the conclusions drawn from such a mapping are only as rigorous as the underlying studies. Bias in the publishing of positive results is well documented (Esarey & Wu, 2016;Franco, Malhotra, & Simonovits, 2014). The authors recognise the possibility of bias, noting in their mapping a 'tendency to share positive results' and 'very few instances of negative impact' (Goodwin & Maru, 2017, p. 168).…”

Legal empowerment and group-based inequalilty

“…More critically, it is worth noting inter alia that the conclusions drawn from such a mapping are only as rigorous as the underlying studies. Bias in the publishing of positive results is well documented (Esarey and Wu 2016;Franco et al 2014). The authors recognize the possibility of bias, noting in their mapping a 'tendency to share positive results' and 'very few instances of negative impact' (Goodwin and Maru 2017: 168).…”