Selective Hypothesis Reporting in Psychology: Comparing Preregistrations and Corresponding Publications

Abstract

This study assesses the extent of selective hypothesis reporting in psychological research by comparing the hypotheses found in a set of 459 preregistrations to the hypotheses found in the corresponding papers. We found that more than half of the preregistered studies we assessed contain omitted hypotheses (N = 224; 52.2%) or added hypotheses (N = 227; 56.8%), and about one-fifth of studies contain changed hypotheses (N = 82; 19%). We found only a small number of studies with demoted hypotheses (N = 2; 1%) and no studies with promoted hypotheses. In all, 59% of studies include at least one hypothesis in one or more of these categories, indicating a substantial bias in presenting and selecting hypotheses by researchers and/or reviewers/editors. Contrary to our expectations, we found that added hypotheses and changed hypotheses were not more likely to be statistically significant than non-selectively reported hypotheses. For the other types of selective hypothesis reporting, no powerful test of the relationship with statistical significance could be carried out. Finally, we found that replication studies were less likely to include selectively reported hypotheses than original studies. Thus, selective hypothesis reporting is problematically common in psychological research and may partly be explained by the fact that authors do not specifically and clearly enough formulate hypotheses in preregistrations and papers. We urge researchers, reviewers, and editors to ensure that hypotheses outlined in preregistrations are clearly formulated and accurately presented in the corresponding papers.

Risk, reproducibility, and reproduction : Essays on scholar’s analytic decisions and consumers' product purchases

“…All the methods used in this thesis have been put in place, to the best of my knowledge, to protect against false positives. There is no doubt that more methodological work is still needed, as it has already been shown that solutions, such as pre-registrations, might not be the solution to all problems (Van den Akker et al, 2022). Nonetheless, the results from Paper I in this thesis show that pre-registration is an important step, as it binds the researcher not to do multiple testing.…”

“…There has been some debate over the use of these pre-registration plans (Pham & Oh, 2021), but there seems to be an overall movement toward using these more (Simmons et al, 2021). It is, however, important to keep discussing these methods, as the first generation of pre-registered analysis still seems to contain errors (Claesen et al, 2021), and even in some cases, there was a massive difference between the published article and the preregistration ( Van den Akker et al, 2022).…”

Big little lies: a compendium and simulation ofp-hacking strategies

“…Ideally, a preregistration provides a detailed overview of all planned analysis steps that can be referenced in a publication to clearly distinguish confirmatory from exploratory analyses [20]. However, analysis plans laid out in preregistrations are often vague, analyses are quietly changed or incompletely reported in publications, and sometimes preregistrations are even written retrospectively [109][110][111]. All these aspects clearly limit the value of preregistrations in practice and spawned the development of registered reports.…”

On the scope of scientific hypotheses

“…There have been recommendations for simpler hypotheses in psychology to avoid misinterpretations and misspecifications [15]. Finally, several evaluations of preregistration practices have found that a significant proportion of articles do not abide by their stated hypothesis or add additional hypotheses [11,[16][17][18]. In sum, while multiple efforts exist to improve scientific practice, our hypothesis-making could improve.…”

“…We consider this useful and timely as, with preregistrations becoming more frequent, the hypothesis-making process is now open and explicit. However, preregistrations are difficult to write [19], and preregistered articles can change or omit hypotheses [11] or they are vague and certain degrees of freedom hard to control for [16][17][18]. One suggestion has been to do less confirmatory research [7,20].…”