Robustness of evidence reported in preprints during peer review

Nelson, Lindsay; Ye, Honghan; Schwenn, Anna; Lee, Shin‐Hyo; Arabi, Salsabil; Hutchins, B. Ian

doi:10.1016/s2214-109x(22)00368-0

Cited by 25 publications

(15 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We observed four preprinted articles that were posted to a preprint server after being published in an accepting journal. This is consistent with previous analyses [3,6,36], which found that preprinting does not always come first but actually occurs throughout the cycle of science communication. Thus, without accounting for the sequence of these events, reported citations, views, and Altmetric scores for each preprinted article may not be caused by preprinting and will tend to overestimate the benefit of amplification associated with preprinting.…”

Section: Limitationssupporting

confidence: 93%

Characterization and Reach of Orthopaedic Research Posted to Preprint Servers: Are We “Undercooking” Our Science?

Rizk

Arza

Jella

et al. 2023

Clin Orthop Relat Res

View full text Add to dashboard Cite

Background Although biomedical preprint servers have grown rapidly over the past several years, the harm to patient health and safety remains a major concern among several scientific communities. Despite previous studies examining the role of preprints during the Coronavirus-19 pandemic, there is limited information characterizing their impact on scientific communication in orthopaedic surgery. Questions/purposes (1) What are the characteristics (subspecialty, study design, geographic origin, and proportion of publications) of orthopaedic articles on three preprint servers? (2) What are the citation counts, abstract views, tweets, and Altmetric score per preprinted article and per corresponding publication? Methods Three of the largest preprint servers (medRxiv, bioRxiv, and Research Square) with a focus on biomedical topics were queried for all preprinted articles published between July 26, 2014, and September 1, 2021, using the following search terms: “orthopaedic,” “orthopedic,” “bone,” “cartilage,” “ligament,” “tendon,” “fracture,” “dislocation,” “hand,” “wrist,” “elbow,” “shoulder,” “spine,” “spinal,” “hip,” “knee,” “ankle,” and “foot.” Full-text articles in English related to orthopaedic surgery were included, while nonclinical studies, animal studies, duplicate studies, editorials, abstracts from conferences, and commentaries were excluded. A total of 1471 unique preprints were included and further characterized in terms of the orthopaedic subspecialty, study design, date posted, and geographic factors. Citation counts, abstract views, tweets, and Altmetric scores were collected for each preprinted article and the corresponding publication of that preprint in an accepting journal. We ascertained whether a preprinted article was published by searching title keywords and the corresponding author in three peer-reviewed article databases (PubMed, Google Scholar, and Dimensions) and confirming that the study design and research question matched. Results The number of orthopaedic preprints increased from four in 2017 to 838 in 2020. The most common orthopaedic subspecialties represented were spine, knee, and hip. From 2017 to 2020, the cumulative counts of preprinted article citations, abstract views, and Altmetric scores increased. A corresponding publication was identified in 52% (762 of 1471) of preprints. As would be expected, because preprinting is a form of redundant publication, published articles that are also preprinted saw greater abstract views, citations, and Altmetric scores on a per-article basis. Conclusion Although preprints remain an extremely small proportion of all orthopaedic research, our findings suggest that nonpeer-reviewed, preprinted orthopaedic articles are being increasingly disseminated. These preprinted articles have a smaller academic and public footprint than their published counterparts, but they still reach a substantial audience through infrequent and superficial online interactions, which are far from equivalent to the engagement facilitated by peer review. Furthermore, the sequence of preprint posting and journal submission, acceptance, and publication is unclear based on the information available on these preprint servers. Thus, it is difficult to determine whether the metrics of preprinted articles are attributable to preprinting, and studies such as the present analysis will tend to overestimate the apparent impact of preprinting. Despite the potential for preprint servers to function as a venue for thoughtful feedback on research ideas, the available metrics data for these preprinted articles do not demonstrate the meaningful engagement that is achieved by peer review in terms of the frequency or depth of audience feedback. Clinical Relevance Our findings highlight the need for safeguards to regulate research dissemination through preprint media, which has never been shown to benefit patients and should not be considered as evidence by clinicians. Clinician-scientists and researchers have the most important responsibility of protecting patients from the harm of potentially inaccurate biomedical science and therefore must prioritize patient needs first by uncovering scientific truths through the evidence-based processes of peer review, not preprinting. We recommend all journals publishing clinical research adopt the same policy as Clinical Orthopaedics and Related Research ®, The Bone & Joint Journal, The Journal of Bone and Joint Surgery, and the Journal of Orthopaedic Research, removing any papers posted to preprint servers from consideration.

show abstract

Section: Limitationssupporting

confidence: 93%

Characterization and Reach of Orthopaedic Research Posted to Preprint Servers: Are We “Undercooking” Our Science?

Rizk

Arza

Jella

et al. 2023

Clin Orthop Relat Res

View full text Add to dashboard Cite

show abstract

“…Among the preprints in medRxiv, 77% (1,077 out of 1,399) were published in peer-reviewed journals within a median of 6 months after posting [14]. In a journal article that assessed how COVID-19 evidence presented in preprints changed after review, point estimate values were found to have changed by an average of 6%, with a strong correlation (0.99) between estimate values before and after review [15]. Although a substantial portion of preprints were later published as peer-reviewed journal articles, and the robustness of the evidence reported in preprints was found to be high, a preprint is preferably cited as a reference after the corresponding peer-reviewed journal article has been published, because the peer review process reduces uncertainty and substantiates the evidence reported in the preprint.…”

Section: Is a Preprint Citable As A Reference In The Manuscript?mentioning

confidence: 99%

Current concerns on journal article with preprint: Korean Journal of Internal Medicine perspectives

Oh¹,

Kang²

2023

Korean J Intern Med

View full text Add to dashboard Cite

Preprints are preliminary research reports that have not yet been peer-reviewed. They have been widely adopted to promote the timely dissemination of research across many scientific fields. In August 1991, Paul Ginsparg launched an electronic bulletin board intended to serve a few hundred colleagues working in a subfield of theoretical high-energy physics, thus launching arXiv, the first and largest preprint platform. Additional preprint servers have since been implemented in different academic fields, such as BioRxiv (2013, Biology; www.biorxiv.org) and medRxiv (2019, Health Science; www.medrxiv.org). While preprint availability has made valuable research resources accessible to the general public, thus bridging the gap between academic and non-academic audiences, it has also facilitated the spread of unsupported conclusions through various media channels. Issues surrounding the preprint policies of a journal must be addressed, ultimately, by editors and include the acceptance of preprint manuscripts, allowing the citation of preprints, maintaining a double-blind peer review process, changes to the preprint's content and authors' list, scoop priorities, commenting on preprints, and preventing the influence of social media. Editors must be able to deal with these issues adequately, to maintain the scientific integrity of their journal. In this review, the history, current status, and strengths and weaknesses of preprints as well as ongoing concerns regarding journal articles with preprints are discussed. An optimal approach to preprints is suggested for editorial board members, authors, and researchers.

show abstract

“…Based on this comparison of preprint posting and article acceptance dates, up to around 20% of preprints could in principle be accepted versions. As we noted in our Discussion, 1 our study was not a measurement of the marginal effects of peer review on manuscripts, but rather a measurement of the expected change from preprint to publication version, because it is not knowable how much peer review (either for gatekeeping at journals or for improvement from informal peer feedback) has occurred before preprint posting. Nor is it knowable how much observed change was driven by reviewer feedback versus being author-initiated.…”

mentioning

confidence: 95%

“…Accordingly, we matched preprints to keywords related to four epidemiological measurements: COVID-19 incidence, infection fatality rate, case fatality rate, and R 0 . 1 Preprinted data were highly stable compared with their published versions. We concluded that our findings, alongside others’, support the use of evidence reported in preprints in decision making, as one component of the biomedical literature.…”

mentioning

confidence: 99%