The trustworthiness and impact of trial preprints for COVID-19 decision-making: A methodological study

Abstract

Purpose: To assess the trustworthiness and impact of preprint trial reports during the COVID-19 pandemic. Data sources: WHO COVID-19 database and the L-OVE COVID-19 platform by the Epistemonikos Foundation (up to August 3rd, 2021) Design: We compare the characteristics of COVID-19 trials with and without preprints, estimate time to publication of COVID-19 preprint reports, describe discrepancies in key methods and results between preprint and published trial reports, report the number of retracted preprints and publications, and assess whether including versus excluding preprint reports affects meta-analytic estimates and the certainty of evidence. For the effects of eight therapies on mortality and mechanical ventilation, we performed meta-analyses including preprints and excluding preprints at 1 month, 3 months, and 6 months after the first trial addressing the therapy became available either as a preprint or publication (120 meta-analyses in total). Results: We included 356 trials, 101 of which are only available as preprints, 181 as journal publications, and 74 as preprints first and subsequently published in journals. Half of all preprints remain unpublished at six months and a third at one year. There were few important differences in key methods and results between trial preprints and their subsequent published reports. We identified four retracted trials, three of which were published in peer-reviewed journals. With two exceptions (2/60; 3.3%), point estimates were consistent between meta-analyses including versus excluding preprints as to whether they indicated benefit, no appreciable effect, or harm. There were nine comparisons (9/60; 15%) for which the rating of the certainty of evidence differed when preprints were included versus excluded, for four of these comparisons the certainty of evidence including preprints was higher and for five of these comparisons the certainty of evidence including preprints was lower. Limitations: The generalizability of our results is limited to COVID-19. Preprints that are subsequently published in journals may be the most rigorous and may not represent all trial preprints. Conclusion: We found no compelling evidence that preprints provide less trustworthy results than published papers. We show that preprints remain the only source of findings of many trials for several months, for a length of time that is unacceptable in a health emergency. We show that including preprints may affect the results of meta-analyses and the certainty of evidence. We encourage evidence users to consider data from preprints in contexts in which decisions are being made rapidly and evidence is being produced faster than can be peer-reviewed.

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

“…Third, for both high-and low-quality preprints, the checklist highlighted issues with transparency and research integrity. This mirrors the state of the COVID-19 literature, as we have seen that though the credibility of the research reported in preprints can be sound (32)(33)(34), data and materials sharing may nonetheless often be lacking (18,19). This could be taken as a weakness of the checklist, as this item is not particularly discriminative of preprint quality.…”

“…Many preprints actually report their results in a balanced way so as not to 'oversell' their findings (31). Preprints on clinical trials for example were shown to report results that may be just as credible as those of publications (32). This finding is reinforced by comparisons between preprints and publications on metrics like the basic reproduction number, transmissibility, incubation period, and infectious period, showing no evidence for a difference in the results presented in peer-reviewed vs. non-peer-reviewed literature (33,34).…”

A guide to preprinting for early-career researchers

“…A majority of the manuscripts posted as preprints go on to be published in a journal; a study of bioRxiv preprints found that two thirds of the preprints appeared at a journal within 2 years ( Abdill and Blekhman , 2019 ). Additional studies that have evaluated the content of preprints and their associated journal publications found that the reporting quality in preprints is within a similar range as that of peer-reviewed articles ( Carneiro et al, 2020 ) and that the main content and conclusions changed little between the preprint and the journal publication for the same work ( Brierley et al, 2022 ; Nicholson et al, 2022 ; Zeraatkar et al, 2022 ). These studies suggest that there is no evidence to consider research findings reported via preprints as less trustworthy than journal publications.…”