Star Wars: The Empirics Strike Back

Abstract

Journals favor rejection of the null hypothesis. This selection upon tests may distort the behavior of researchers. Using 50, 000 tests published between 2005 and 2011 in the AER, JPE, and QJE, we identify a residual in the distribution of tests that cannot be explained by selection. The distribution of p-values exhibits a two humped camel shape with abundant p-values above 0.25, a valley between 0.25 and 0.10, and a bump slightly below 0.05. The missing tests (with p-values between 0.25 and 0.10) can be retrieved just after the 0.05 threshold and represent 10% to 20% of marginally rejected tests. Our interpretation is that researchers might be tempted to inflate the value of those just-rejected tests by choosing a "significant" specification. We propose a method to measure this residual and describe how it varies by article and author characteristics.

Detecting p-hacking

“…We can see that, depending on π, the t-curve takes many different forms. This simple numerical example further demonstrates that the distribution of alternatives can induce humps around 1.96, as documented empirically for instance by Gerber and Malhotra (2008), Brodeur et al (2016Brodeur et al ( , 2018 and Vivalt (2019), even if there is no phacking. Thus, humps generated by p-hacking cannot be distinguished from humps generated by the distribution of alternatives, which suggests that testing for p-hacking based on the shape of the t-curve around 1.96 (or any other significance threshold) can be problematic.…”

“…On the one hand, it is often plausible to assume that the conditional publication probability is decreasing in p (i.e., more significant results are more likely to get published), which implies that g S=1 is non-increasing under the assumptions of Theorem 1. In Appendix A, we present a simple reduced form model in the spirit of Brodeur et al (2016), which provides a formal justification for a decreasing publication probability. 5 In this case, g S=1 is non-increasing whenever g is non-increasing.…”

“…A researchers' ability to explore various ways of analyzing and manipulating data and then selectively report the ones that yield statistically significant results, commonly referred to as p-hacking, undermines the scientific credibility of reported results. To assess the extent of this problem researchers have begun to examine distributions of t-statistics (t-curves) and p-values (p-curves) across studies in various fields, with mixed results (e.g., Bishop and Thompson, 2016;Brodeur et al, 2016Brodeur et al, , 2018Gerber and Malhotra, 2008;Head et al, 2015;Jager and Leek, 2013;Simonsohn et al, 2014;Vivalt, 2019). This paper examines more closely what can be learned from these distributions, and whether or not these tests are likely to be informative about the extent to which p-hacking occurs.…”

“…We apply the tests to assess the prevalence of p-hacking in economics and other disciplines based on two large datasets of p-values. The first dataset, collected by Brodeur et al (2016), contains test statistics and p-values from papers published in the American Economic Review, the Quarterly Journal of Economics, and the Journal of Political Economy between 2005 and 2011.…”

Why and how systematic strategies decay

“…This paper is mostly a contribution to the rising literature on robustness of findings in economics in general. An earlier literature documented the accumulation of empirical results around key thresholds of statistical significance, as an evidence that many empirical results are fitted to "work" in-sample (see Brodeur et al (2016) for a recent reference). Some papers advocate or develop methods to deal with multiple testing problems (see for instance Harvey et al (2015), Yan and Zheng (2017) or Giglio et al (2020) for recent work).…”