The Problem of New Evidence: P-Hacking and Pre-Analysis Plans

Abstract

We provide a novel articulation of the epistemic peril of p-hacking using three resources from philosophy: predictivism, Bayesian confirmation theory, and model selection theory. We defend a nuanced position on p-hacking: p-hacking is sometimes, but not always, epistemically pernicious. Our argument requires a novel understanding of Bayesianism, since a standard criticism of Bayesian confirmation theory is that it cannot represent the influence of biased methods. We then turn to pre-analysis plans, a methodological device used to mitigate p-hacking. Some say that pre-analysis plans are epistemically meritorious while others deny this, and in practice pre-analysis plans are often violated. We resolve this debate with a modest defence of pre-analysis plans. Further, we argue that pre-analysis plans can be epistemically relevant even if the plan is not strictly followed—and suggest that allowing for flexible pre-analysis plans may be the best available policy option.

Preregistration: Definition, Advantages, Disadvantages, and How It Can Help Against Questionable Research Practices

“…Dating back from the time of Sir Karl Popper (e.g., Popper, 2005), a test is argued to be severe when it is strong enough to falsify a theory. In this line of reasoning, a preregistration allows others to evaluate whether a performed test was capable of falsifying a tested theory (Hitzig & Stegenga, 2020;Lakens, 2020;O'Donohue, 2021;Vanpaemel, 2019).…”

Preregistration in diverse contexts: a preregistration template for the application of cognitive models

“…the debate in philosophy of science surrounding prediction and accommodation of evidence. The overall consensus of this debate is that accommodated evidence/postdiction can sometimes be epistemologically equivalent to prediction ( [71]; for empirical evidence supporting this consensus: [72]).…”

Red herrings about relative measures: A response to Hoefer and Krauss

“…What do these issues have to do with outcome measures? Nevertheless, I have spent a good part of my career discussing these and related issues (see, e.g., Stegenga et al, 2017;Hitzig & Stegenga, 2020;Moynihan et al, 2019).…”