Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment

Abstract

Brodeur, Cook, and Heyes (2020) study hypothesis tests from economic articles and find evidence for p-hacking and publication bias, in particular for instrumental variable and difference-in-difference studies. When adjusting for rounding errors (introducing a novel method), statistical evidence for p-hacking from randomization tests and caliper tests at the 5 percent significance threshold vanishes for difference-in-differnce studies but remains for instrumental variable studies. Results at the 1 percent and 10 percent significance thresholds remain largely similar. In addition, Brodeur, Cook, and Heyes derive latent distributions of z-statistics absent publication bias using two different approaches. We establish for each approach a result that challenges its applicability. (JEL A14, C12, C52)

Contexts of Convenience: Generalizing from Published Evaluations of School Finance Policies

“…One type of bias, known as ‘p-hacking’, occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant.” Recent analysis, albeit not without controversy, attempts to quantify the extent of p-hacking in economics. The magnitude of it is under question, but its existence seems indisputable (Brodeur et al (2020, 2022); Kranz and Pütz (2022); Brodeur et al (2023)). 14 Thus, the subsequent reporting of statistical significance (and impact parameters themselves) may be directly influenced by the research bias induced by the publication process.…”

Preregistration and Credibility of Clinical Trials*

“…A potential concern in the literature that analyzes the distribution of z-scores is that rounding of parameter estimates, standard errors, test statistics, or p-values could influence the estimation of densities and discontinuity tests, as rounding might lead to the accumulation of mass at specific values. For instance, papers that construct z-scores directly from parameter estimates and standard errors in published articles are concerned about rounding leading to many z-scores of exactly 2, right above the significance threshold of 1.96, which might actually be below 1.96 if reported more precisely Kranz and P ütz, 2022). To test for the robustness of estimates with respect to such rounding issues, de-rounding procedures have been proposed.…”

Beauty and Professional Success: A Meta-Analysis

“…In other words, we face a classical endogeneity problem in our meta-analysis specifications. & Putz, 2022). For this reason we prefer the results reported in Panel A of Table B3 to the precision-weighted specifications reported in Panel B.…”