Comparison of Clinical Study Results Reported in medRxiv Preprints vs Peer-reviewed Journal Articles

Abstract

ImportancePreprints have been widely adopted to enhance the timely dissemination of research across many scientific fields. Concerns remain that early, public access to preliminary medical research has the potential to propagate misleading or faulty research that has been conducted or interpreted in error.ObjectiveTo evaluate the concordance among study characteristics, results, and interpretations described in preprints of clinical studies posted to medRxiv that are subsequently published in peer-reviewed journals (preprint-journal article pairs).Design, Setting, and ParticipantsThis cross-sectional study assessed all preprints describing clinical studies that were initially posted to medRxiv in September 2020 and subsequently published in a peer-reviewed journal as of September 15, 2022.Main Outcomes and MeasuresFor preprint-journal article pairs describing clinical trials, observational studies, and meta-analyses that measured health-related outcomes, the sample size, primary end points, corresponding results, and overarching conclusions were abstracted and compared. Sample size and results from primary end points were considered concordant if they had exact numerical equivalence.ResultsAmong 1399 preprints first posted on medRxiv in September 2020, a total of 1077 (77.0%) had been published as of September 15, 2022, a median of 6 months (IQR, 3-8 months) after preprint posting. Of the 547 preprint-journal article pairs describing clinical trials, observational studies, or meta-analyses, 293 (53.6%) were related to COVID-19. Of the 535 pairs reporting sample sizes in both sources, 462 (86.4%) were concordant; 43 (58.9%) of the 73 pairs with discordant sample sizes had larger samples in the journal publication. There were 534 pairs (97.6%) with concordant and 13 pairs (2.4%) with discordant primary end points. Of the 535 pairs with numerical results for the primary end points, 434 (81.1%) had concordant primary end point results; 66 of the 101 discordant pairs (65.3%) had effect estimates that were in the same direction and were statistically consistent. Overall, 526 pairs (96.2%) had concordant study interpretations, including 82 of the 101 pairs (81.2%) with discordant primary end point results.Conclusions and RelevanceMost clinical studies posted as preprints on medRxiv and subsequently published in peer-reviewed journals had concordant study characteristics, results, and final interpretations. With more than three-fourths of preprints published in journals within 24 months, these results may suggest that many preprints report findings that are consistent with the final peer-reviewed publications.

Phase 1 of the National Institutes of Health Preprint Pilot: Testing the viability of making preprints discoverable in PubMed Central and PubMed

“…This may be, in part, because NLM limited preprint collecting activities to those that report NIH support and, therefore, are subject to the NIH grant selection peer review process [26] or internal approval processes. NLM's experience with Phase 1 is also consistent with results reported in a growing body of literature comparing the content of articles posted as preprints to the content of the same article following publication in a peer-reviewed journal [27][28][29][30]…”

Describing the landscape of medical education preprints on medRxiv

“…Should preprints be an evidence source when no peer-reviewed evidence is available, especially if and when researchers find high concordance between preprints and their eventual peer-reviewed counterpart? 14,45,47 These and many other questions about preprints and their use should be collectively answered by the medical education community.…”

Comparison of COVID-19 Preprint and Peer-Reviewed Versions of Studies on Therapies for Critically Ill Patients

“…These results are largely in line with prior studies evaluating the content of both COVID-19 and non-COVID-19 preprint and peer-reviewed literature. 31,32,33 These studies found that changes in text were minimal, with discordance in primary outcomes occurring in 20 to 30% of articles. Based on the time from preprint availability to publication, this may suggest that the peer review process may take at least 6 weeks, and in most cases at least 16 (although it is important to note that we did not have time of initial journal submission).…”

“…This is consistent with other published studies that have performed a similar analysis, although not specifically on articles regarding COVID-19. 31,33 It should be noted that though there is a difference in the effect size of the primary outcome, this was often minimal and did not affect statistical significance. Our data do suggest that secondary outcomes should be interpreted with AC (n = 2) 10,11 D (n = 5) [12][13][14][15][16] HCQ (n = 5) [17][18][19][20][21] RDV (n = 4) [22][23][24][25] T (n = 5) [26][27][28][29][30] Median days from preprint to publication, (IQR) 147. caution and may not be reliable in the clinical decision-making process until peer-review has been completed, resulting in final publication.…”