Changes in evidence for studies assessing interventions for COVID-19 reported in preprints: meta-research study

Abstract

Background The increasing use of preprints to disseminate evidence on the effect of interventions for the coronavirus disease 2019 (COVID-19) can lead to multiple evidence sources for a single study, which may differ in the reported evidence. We aim to describe the proportion of evidence on the effect of interventions for COVID-19 from preprints and journal articles and map changes in evidence between and within different sources reporting on the same study. Methods Meta-research study. We screened the Cochrane living systematic review and network meta-analysis (COVID-NMA) database to identify all preprints and journal articles on all studies assessing interventions for COVID-19 published up to 15 August 2020. We compared all evidence sources (i.e., preprint and associated journal article) and the first and latest versions of preprints for each study to identify changes in two evidence components: study results (e.g., numeric change in hazard ratio, odds ratio, event rate, or change in p value > or < 0.05 in any outcome) and abstract conclusions (classified as positive, negative or neutral regarding the intervention effect, and as reporting uncertainty in the findings or not). Changes in study results were further classified as important changes if they (1) represented a change in any effect estimate by ≥ 10% and/or (2) led to a change in the p value crossing the threshold of 0.05. Results We identified 556 studies. In total, 338 (61%) had been reported in a preprint: 66 (20%) of these had an associated journal article (median time to publication 76 days [interquartile range (IQR) 55–106]) and 91 (27%) had > 1 preprint version. A total of 139 studies (25% of the overall sample) were reported in multiple evidence sources or versions of the same source: for 63 (45%), there was a change in at least one evidence component between or within sources (42 [30%] had a change in study results, and in 29 [21%] the change was classified as important; 33 [24%] had a change in the abstract conclusion). For studies with both a preprint and an article, a median of 29% (IQR 14–50) of total citations were attributed to the preprint instead of the article. Conclusions Results on the effect of interventions for COVID-19 are often reported in multiple evidence sources or source versions for a single study. Evidence is not stable between and within evidence sources. Real-time linkage of all sources per study could help to keep systematic reviews up-to-date.

Using preprints in evidence synthesis: Commentary on experience during the COVID-19 pandemic

“…The majority of COVID-19 preprints have been found to have a single version, with some preprints existing beyond two versions. [ 6 , 15 ] A single DOI may be retained for all versions. [4] No standard for a new version requirement was identified, thus new versions may cover varying levels of change and substantial changes between preprint versions have been identified.…”

“…[ 4 , 5 ] The majority of preprint servers provide a DOI for each manuscript. [4] Manuscripts will remain on the servers and while up to a third will later be published in peer-reviewed journals (resulting in multiple versions of the same study [6] ), some may never be submitted for peer review or be accepted. 6 , 7 , 8 , 9 The advantages of preprints include early and rapid dissemination, opportunities for informal commenting, potential decreased publication bias and greater recognition and visibility of work, particularly for early-career researchers.…”

“… [4] Manuscripts will remain on the servers and while up to a third will later be published in peer-reviewed journals (resulting in multiple versions of the same study [6] ), some may never be submitted for peer review or be accepted. 6 , 7 , 8 , 9 The advantages of preprints include early and rapid dissemination, opportunities for informal commenting, potential decreased publication bias and greater recognition and visibility of work, particularly for early-career researchers. 10 , 11 , 12 Evidence also suggests that peer-reviewed articles with a co-existing preprint are associated with more attention and citations than those without a preprint.…”

A metaresearch study revealed susceptibility of Covid-19 treatment research to white hat bias: first, do no harm

“…The need of rapid knowledge dissemination has led to a surge of fast-track pre-prints, resulting in a complex network of academic and non-peer-reviewed publications [2] . The situation is further complicated by the existence of multiple versions of a single study in open archives and as journal articles, often presenting contradictory outcomes [3] . As a result, the process of decision-making and identification of credible evidence remains a challenge both for the researcher and the clinician.…”

A living meta-ecological study of the consequences of the COVID-19 pandemic on mental health

“…Another study appeared to show that the use of masks is useless, but was retracted later because the authors did not interpret their data correctly [ 8 ]. Differences between preprints and associated journal articles have also been reported ( 9 ). In our field, many teams have conducted cross-sectional online surveys to find out whether the pandemic led to an increase in mental health problems.…”