Reproducibility of COVID-19 pre-prints

Collins, Annie; Alexander, Rohan

doi:10.1007/s11192-022-04418-2

Cited by 10 publications

(7 citation statements)

References 24 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study we verified previous code transparency results [10, 11], finding that less than 20% of randomly selected infectious disease modelling studies provide working links to code. Notably, the top-cited set of studies released code far more frequently (48%); however this is may be a function of the journals in which these articles were published and the mandates which they impose.…”

Section: Discussionsupporting

confidence: 86%

“…Recently, Zavalis et al [10] evaluated the transparency of published articles in infectious disease modelling by recording the number of papers that had publicly released accompanying code. They found that few authors (20%) share their code, with Collins et al [11] obtaining similar results for COVID-19 preprints 1 (21-33%). These release rates align with other computational domains such as machine learning (33%) [12] and physics (6%) [13].…”

Section: Introductionmentioning

confidence: 74%

See 1 more Smart Citation

Reproducibility of COVID-era infectious disease models

Henderson,

Hickson,

Furlong

et al. 2023

Preprint

View full text Add to dashboard Cite

Infectious disease modelling has been prominent throughout the COVID-19 pandemic, help- ing to understand the virus' transmission dynamics and inform response policies. Given their potential importance and translational impact, we evaluated the computational reproducibility of infectious disease modelling articles from the COVID era. We found that only four out of 100 randomly sampled studies released between January 2020 and August 2022 could be com- putationally reproduced using the resources provided (e.g., code, data, instructions). For the 100 most highly cited articles from the same period we found that only 11 were reproducible. Reflecting on our experience, we discuss common issues affecting computational reproducibility and how these might be addressed

show abstract

Section: Discussionsupporting

confidence: 86%

Section: Introductionmentioning

confidence: 74%

Reproducibility of COVID-era infectious disease models

Henderson,

Hickson,

Furlong

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…More literature searches identified another 44 eligible reports for inclusion, giving a total of 114 eligible meta-research studies examining a combined total of 2 254 031 primary articles for the review. 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 ...…”

Section: Resultsmentioning

confidence: 99%

“…Supplementary results in the supplementary information provides further information on the individual participant data retrieval process, as well as the outcomes of the data integrity checks. In total, 108 reports of 105 meta-research studies assessing a total of 2 121 580 primary articles were included in the quantitative analysis, 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 ...…”

Section: Resultsmentioning

confidence: 99%

Prevalence and predictors of data and code sharing in the medical and health sciences: systematic review with meta-analysis of individual participant data

Hamilton

Hong

Fraser

et al. 2023

BMJ

View full text Add to dashboard Cite

Objectives To synthesise research investigating data and code sharing in medicine and health to establish an accurate representation of the prevalence of sharing, how this frequency has changed over time, and what factors influence availability. Design Systematic review with meta-analysis of individual participant data. Data sources Ovid Medline, Ovid Embase, and the preprint servers medRxiv, bioRxiv, and MetaArXiv were searched from inception to 1 July 2021. Forward citation searches were also performed on 30 August 2022. Review methods Meta-research studies that investigated data or code sharing across a sample of scientific articles presenting original medical and health research were identified. Two authors screened records, assessed the risk of bias, and extracted summary data from study reports when individual participant data could not be retrieved. Key outcomes of interest were the prevalence of statements that declared that data or code were publicly or privately available (declared availability) and the success rates of retrieving these products (actual availability). The associations between data and code availability and several factors (eg, journal policy, type of data, trial design, and human participants) were also examined. A two stage approach to meta-analysis of individual participant data was performed, with proportions and risk ratios pooled with the Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis. Results The review included 105 meta-research studies examining 2 121 580 articles across 31 specialties. Eligible studies examined a median of 195 primary articles (interquartile range 113-475), with a median publication year of 2015 (interquartile range 2012-2018). Only eight studies (8%) were classified as having a low risk of bias. Meta-analyses showed a prevalence of declared and actual public data availability of 8% (95% confidence interval 5% to 11%) and 2% (1% to 3%), respectively, between 2016 and 2021. For public code sharing, both the prevalence of declared and actual availability were estimated to be <0.5% since 2016. Meta-regressions indicated that only declared public data sharing prevalence estimates have increased over time. Compliance with mandatory data sharing policies ranged from 0% to 100% across journals and varied by type of data. In contrast, success in privately obtaining data and code from authors historically ranged between 0% and 37% and 0% and 23%, respectively. Conclusions The review found that public code sharing was persistently low across medical research. Declarations of data sharing were also low, increasing over time, but did not always correspond to actual sharing of data. The effectiveness of mandatory data sharing policies varied substantially by journal and type of data, a finding that might be informative for policy makers when designing policies and allocating resources to audit compliance. Systematic review registration Open Science Framework doi: 10.17605/OSF.IO/7SX8U .

show abstract

“…Many authors (e.g. Gustot 2020; Sumner et al 2020; Collins and Alexander 2022) have emphasized the importance of reproducing COVID‐19 research and have begun to catalog the availability of code and data within the literature. Geographers have produced similar catalogs of geographical analyses of COVID‐19 but have limited their reviews to listing and categorizing the literature by topical focus and methodological approach (Agbehadji et al 2020; Ahasan et al 2020; Franch‐Pardo et al 2020, 2021).…”

Section: Empirical Context and The Selection Of Studies For Reproductionmentioning

confidence: 99%

A Framework for Moving Beyond Computational Reproducibility: Lessons from Three Reproductions of Geographical Analyses of COVID‐19

Kedron,

Bardin,

Holler

et al. 2023

Geographical Analysis

View full text Add to dashboard Cite

Despite recent calls to make geographical analyses more reproducible, formal attempts to reproduce or replicate published work remain largely absent from the geographic literature. The reproductions of geographic research that do exist typically focus on computational reproducibility—whether results can be recreated using data and code provided by the authors—rather than on evaluating the conclusion and internal validity and evidential value of the original analysis. However, knowing if a study is computationally reproducible is insufficient if the goal of a reproduction is to identify and correct errors in our knowledge. We argue that reproductions of geographic work should focus on assessing whether the findings and claims made in existing empirical studies are well supported by the evidence presented. We aim to facilitate this transition by introducing a model framework for conducting reproduction studies, demonstrating its use, and reporting the findings of three exemplar studies. We present three model reproductions of geographical analyses of COVID‐19 based on a common, open access template. Each reproduction attempt is published as an open access repository, complete with pre‐analysis plan, data, code, and final report. We find each study to be partially reproducible, but moving past computational reproducibility, our assessments reveal conceptual and methodological concerns that raise questions about the predictive value and the magnitude of the associations presented in each study. Collectively, these reproductions and our template materials offer a practical framework others can use to reproduce and replicate empirical spatial analyses and ultimately facilitate the identification and correction of errors in the geographic literature.

show abstract

Reproducibility of COVID-19 pre-prints

Cited by 10 publications

References 24 publications

Reproducibility of COVID-era infectious disease models

Reproducibility of COVID-era infectious disease models

Prevalence and predictors of data and code sharing in the medical and health sciences: systematic review with meta-analysis of individual participant data

A Framework for Moving Beyond Computational Reproducibility: Lessons from Three Reproductions of Geographical Analyses of COVID‐19

Contact Info

Product

Resources

About