Reporting all results efficiently: A RARE proposal to open up the file drawer

Abstract

While the social sciences have made impressive progress in adopting transparent research practices that facilitate verification, replication, and reuse of materials, the problem of publication bias persists. Bias on the part of peer reviewers and journal editors, as well as the use of outdated research practices by authors, continues to skew literature toward statistically significant effects, many of which may be false positives. To mitigate this bias, we propose a framework to enable authors to report all results efficiently (RARE), with an initial focus on experimental and other prospective empirical social science research that utilizes public study registries. This framework depicts an integrated system that leverages the capacities of existing infrastructure in the form of public registries, institutional review boards, journals, and granting agencies, as well as investigators themselves, to efficiently incentivize full reporting and thereby, improve confidence in social science findings. In addition to increasing access to the results of scientific endeavors, a well-coordinated research ecosystem can prevent scholars from wasting time investigating the same questions in ways that have not worked in the past and reduce wasted funds on the part of granting agencies.

Spoiler alert: How narrative film captures attention

“…And although we recognize this tenet, we believe that in limited cases null results can be meaningful. As discussed in Laitin et al (2021), the scientific community is moving towards adopting transparent research practices in order to have a more complete view of performed experiments.…”

“…And although we recognize this tenet, we believe that in limited cases null results can be meaningful. As discussed in Laitin et al (2021), the scientific community is moving towards adopting transparent research practices in order to have a more complete view of performed experiments. Despite these limitations, the current study advances previous research by showing how competing motives (viewing a narrative and maintaining one's goal) involve a type of tension.…”

Seeking Overlapping Neuroanatomical Alterations between Dyslexia and Attention-Deficit/Hyperactivity Disorder: A Meta-Analytic Replication Study

“…Knowing that some attempts to find a given effect of interest have yielded null results is of great relevance when running a CBMA [ 81 , 82 ]. Quantifying the exact number of null experiments is generally hard, as formalized in the so-called “file-drawer effect” bias [ 31 , 81 , 82 ]. Nonetheless, McGrath and Stoodley [ 1 ] identified nine of them during their literature search.…”

Credibility crisis in agricultural economics

“…This selective reporting includes fishing for statistically significant results (also called p‐hacking). Selective reporting also includes failing to submit manuscripts with null results (Laitin et al, 2021) or, within a publication, failing to report statistically nonsignificant results. To find statistically significant effect sizes, authors may analyze multiple treatments or outcome variables, use various regression specifications, choose regressors or identification strategies, and select or exclude samples to get desired results.…”