Null-hacking, a lurking problem

Abstract

Pre-registration of analysis plans involves making data-analysis decisions before the data is run in order to prevent flexibly re-running it until a specific result appears (p-hacking). Just because a model and result is pre-registered, however, does not make it true. The complement to p-hacking, null-hacking, is the use of the same questionable research practices to re-analyze open data to return a null finding. We provide a vocabulary for null-hacking and introduce the threat it poses to open science and pre-registration. Null-hacking forces consideration of model fit to compare pre-registered and ‘alternative’ models. The reason null-hacking cannot be ignored is a null-hacked model can easily provide better fit to the data than a pre-registered one. Model fit, however, is a precarious problem and in the extreme challenges the justifications of pre-registration. Focusing just on model fit by only selecting a ‘best fitting model’ eliminates pre-registration, while giving default preference to pre-registered results ignores how well our models can represent the data. We provide a beginning solution aimed at retaining the advantage and justifications of pre-registration, including model fit, and providing protection against null-hacking. We call this Fully-Informed Model Pre-registration and it involves strict supervised machine learning to maximize local model fit within heavily pre-specified decisions. This solution eliminates exploratory analyses and maximizes local model fit, eliminating the only justifications null-hacked results have. It is not yet a complete solution but merely the groundwork for why other approaches have and will fail and what a solution may look like.

Replicator degrees of freedom allow publication of misleading failures to replicate

“…It is now widely appreciated that original investigators face a conflict between the desire for accuracy and the career incentive to discover statistically significant results. There seems to be a widespread implicit presumption, however, that investigators who undertake replication tests are not subject to similar conflicts, but there are good reasons to believe that they are (16, 17). As replicability and research integrity have become topics of increasing interest, failures to replicate important original findings have begun to be published in top journals (1, 4–6, 18), while successful direct replications of existing findings receive much less attention.…”

“…The one-sided focus on “p-hacking,” the motivated pursuit of statistically significant results by original investigators, ignores (and arguably, contributes to) a new threat to research integrity posed by “null hacking,” the motivated pursuit of null results by replicating investigators (16). The purpose of this article is to demonstrate that replicator degrees of freedom, defined as discretion exercised at 2 stages of the replication process—experimental design and data analysis—can cause replicating investigators to arrive at incorrect conclusions about the replicability of an original finding.…”

No relationship between researcher impact and replication effect: an analysis of five studies with 100 replications

“…There are also a number of reasons why replication efforts may fail to replicate a real effect including lack of power in the replications (Cohen, 1969), lack of fidelity among researchers to the procedures of the original study (see Gilbert et al, 2016b), unacknowledged variance in auxiliary assumptions (Earp & Trafimow, 2015), deliberate questionable research practices used by the replicator to show a lack of evidence (e.g., Protzko, 2018), among others.…”

An advanced method to streamline p-hacking

“…Similarly, higher extra pointless values (e.g., 0.20 to 0.80) can be used to rule out unloved control variables or support a researcher's effort to discredit seminal research findings. Specifically, the ongoing debates on replicability in various fields of sciences (e.g., Ankel-Peters et al, 2023;Camerer et al, 2016;Camerer et al, 2018;Dennis et al, 2020;Nosek et al, 2022;Page et al, 2021) further sparked the need for unambiguously unsuccessful replication studies with p-values far above 0.05 to boost the replicator's reputation as a scientific myth-buster (Bryan et al, 2019;Protzko, 2018). The extra pointless metric will also help with the publication of responses to such criticism that successfully replicate the effect under scrutiny, thereby fueling a never-ending back-andforth of commentaries and rejoinders (i.e., more publications).…”