A meta-epidemiological assessment of transparency indicators of infectious disease models

Abstract

Mathematical models have become very influential, especially during the COVID-19 pandemic. Data and code sharing are indispensable for reproducing them, protocol registration may be useful sometimes, and declarations of conflicts of interest (COIs) and of funding are quintessential for transparency. Here, we evaluated these features in publications of infectious disease-related models and assessed whether there were differences before and during the COVID-19 pandemic and for COVID-19 models versus models for other diseases. We analysed all PubMed Central open access publications of infectious disease models published in 2019 and 2021 using previously validated text mining algorithms of transparency indicators. We evaluated 1338 articles: 216 from 2019 and 1122 from 2021 (of which 818 were on COVID-19); almost a six-fold increase in publications within the field. 511 (39.2%) were compartmental models, 337 (25.2%) were time series, 279 (20.9%) were spatiotemporal, 186 (13.9%) were agent-based and 25 (1.9%) contained multiple model types. 288 (21.5%) articles shared code, 332 (24.8%) shared data, 6 (0.4%) were registered, and 1197 (89.5%) and 1109 (82.9%) contained COI and funding statements, respectively. There was no major changes in transparency indicators between 2019 and 2021. COVID-19 articles were less likely to have funding statements and more likely to share code. Manual assessment of 10% of the articles that were identified by the text mining algorithms as fulfilling transparency indicators showed that 24/29 (82.8%) actually shared code, 29/33 (87.9%) actually shared data; and all had COI and funding statements, but 95.8% disclosed no conflict and 11.7% reported no funding. On manual assessment, 5/6 articles identified as registered had indeed been registered. Transparency in infectious disease modelling is relatively low, especially for data and code sharing. This is concerning, considering the nature of this research and the heightened influence it has acquired.

Transparency in infectious disease research: a meta-research survey of specialty journals

“…This information allowed a correction of the estimates of the proportion of articles that satisfy each of the transparency indicators. As in previous work(11), the corrected proportion C(i) of publications satisfying an indicator i was obtained by U(i) × TP + (1 − U(i)) × FN, where U(i) is the uncorrected proportion detected by the automated algorithm, TP is the proportion of true positives (proportion of those manually verified to satisfy the indicator among those identified by the algorithm as satisfying the indicator), and FN is the proportion of false negatives (proportion of those manually found to satisfy the indicator among those categorized by the algorithm not to satisfy the indicator).…”

“…In a previous evaluation (11) we had studied 1338 articles from 2019 and 2021 and we had showed that approximately a quarter of publications shared code, and a modestly higher proportion shared data in studies of infectious disease modelling. High rates of conflict of interest and funding disclosures were also observed in the same study (around 90% for both).…”

“…In this regard, transparency features, such as sharing of data and code, availability of pre-registered protocols, as well as reporting of conflicts of interest and funding, can be fundamental in evaluating the evidence obtained by research investigations in infectious diseases (8)(9)(10). Previous work has assessed these transparency indicators in depth in infectious disease models specifically, a type of research that became highly popular and influential during the COVID-19 pandemic (11). It was found that the majority of such epidemiological modeling studies do not share their data and/or their code, and very few have registered protocols.…”

Rates and predictors of data and code sharing in the medical and health sciences: A systematic review with meta-analysis of individual participant data

“…Of the remaining 216 articles, full-text articles were retrieved for all papers, and 70 were adjudicated as eligible for the review. Furthermore, the additional searches revealed another 44 eligible reports for inclusion, resulting in a total of 114 eligible meta-research studies examining a combined total of 2,254,031 primary articles for the review [8][9][10][11][12][13][14][15].…”

COVID-19 models and expectations – Learning from the pandemic

“…Perhaps they do not even deserve to be seen as “studies”, but more as semi-formal speculations. This does not mean, however, that models should be discarded or that there is no room for improvement in terms of their transparency and validation ( Zavalis and Ioannidis, 2022 ). Medley (2022) presents an insider view of the difficulties and challenges that arose as models had to be produced, run, interpreted and applied in the UK in real time under critical circumstances.…”