Robustness of evidence reported in preprints during peer review

Abstract

Adoption of preprints dramatically expanded during the COVID-19 pandemic. Many have expressed concern that the risk of flawed decision-making is increased by relying on preprint data that would not survive peer review. We therefore asked how much the information presented in preprints is expected to change after review. We quantify attrition dynamics of over 1000 epidemiological estimates first reported in 100 matched preprints studying COVID-19. We find that 89% of point estimates persist through peer review. Of these, the correlation between preprint and published estimate values is extremely high at 0.99, and there is no systematic trend toward estimate inflation or deflation during review. A higher degree of data alteration during peer review, either in terms of magnitude or deletion, might be expected in papers never published because of their lower quality, which could limit the generalizability of our results. Importantly, we find that expert peer review scores of preprint quality are not related to eventual publication in a peer reviewed journal, mitigating this concern. Uncertainty is reduced somewhat, however, as authors add another 18% of data points compared to the preprint version. Confidence interval ranges also decrease by a small but statistically significant 7%. Therefore, the evidence base presented in preprints is highly stable, and where data change during review, uncertainty is expected to decrease by a small amount on average. These results lend credence to the use of preprints, as one component of the biomedical research literature, in decision-making. These results can help inform the use of preprints during the ongoing pandemic as well as future disease outbreaks.

Predicting causal citations without full text

“…In addition, analysis of the primary data constituting the evidence base suggests that estimate values in published papers are nearly identical to those in preprint versions (24, 25). In addition, there are not measurable differences in peer review evaluations of article quality between preprints that are published vs. those that remain unpublished (25). These studies support the use of preprints, as one important component of the scientific literature, in downstream research development and decision-making.…”

“…Preprints describing clinical trials on COVID-19 did not change their conclusions during peer review (24). Finally, primary data from preprints are remarkably stable as they undergo peer review (25).…”

“…Comparison of preprints and their peer-reviewed versions on COVID-related research also found coherence in the reported results (41,44,46), extending the findings of these studies. In addition, analysis of the primary data constituting the evidence base suggests that estimate values in published papers are nearly identical to those in preprint versions (24,25). In addition, there are not measurable differences in peer review evaluations of article quality between preprints that are published vs. those that remain unpublished (25).…”

“…Preprints that have been published were identified through the Europe PMC application programming interface, which matches the Digital Object Identifier (DOI) of the preprint and published versions of a paper. DOIs corresponding to bioRxiv preprints were identified and their full text, which includes a structured reference list, were downloaded in October 2020 (25). References were matched with the National Library of Medicine's Hydra citation resolution service (4).…”