An evaluation of prospective COVID-19 modelling studies in the USA: from data to science translation

Nixon, Kristen; Jindal, Sonia; Parker, Felix; Reich, Nicholas G.; Ghobadi, Kimia; Lee, Elizabeth C.; Truelove, Shaun; Gardner, Lauren

doi:10.1016/s2589-7500(22)00148-0

Cited by 24 publications

(17 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here, we focus on reported cases and primarily on the European Forecast Hub but our observations hold, in our view, across COVID-19 Forecast Hubs and to a lesser degree targets. We focus on reported cases as these represent the most common forecast target for COVID-19 forecast models (Nixon et al 2022), they are often of the most direct interest due to being a leading indicator for other metrics such as hospitalisations (Meakin et al 2022), and they are generally the most challenging to predict (Sherratt et al 2022). In general, 5 main classes of forecast models are submitted (Bracher et al 2022; Cramer et al 2022), statistical forecasting models such as ARIMA models, mechanistic forecasting models based on the compartmental modelling framework and its generalisations (Srivastava, Xu, and Prasanna 2020; Li et al 2021), semi-mechanistic approaches that blend both of these approaches (Castro et al 2021; Bosse et al 2022), agent-based simulation models (Rakowski et al 2010; Adamik et al 2020), and human insight based forecast models that may also include elements of other methods (Karlen 2020; Bosse et al 2022).…”

Section: Methodsmentioning

confidence: 99%

Evaluating an epidemiologically motivated surrogate model of a multi-model ensemble

Abbott

Sherratt

Bosse

et al. 2022

Preprint

View full text Add to dashboard Cite

Multi-model and multi-team ensemble forecasts have become widely used to generate reliable short-term predictions of infectious disease spread. Notably, various public health agencies have used them to leverage academic disease modelling during the COVID-19 pandemic. However, ensemble forecasts are difficult to interpret and require extensive effort from numerous participating groups as well as a coordination team. In other fields, resource usage has been reduced by training simplified models that reproduce some of the observed behaviour of more complex models. Here we used observations of the behaviour of the European COVID-19 Forecast Hub ensemble combined with our own forecasting experience to identify a set of properties present in current ensemble forecasts. We then developed a parsimonious forecast model intending to mirror these properties. We assess forecasts generated from this model in real time over six months (the 15th of January 2022 to the 19th o July 2022) and for multiple European countries. We focused on forecasts of cases one to four weeks ahead and compared them to those by the European forecast hub ensemble. We find that the surrogate model behaves qualitatively similarly to the ensemble in many instances, though with increased uncertainty and poorer performance around periods of peak incidence (as measured by the Weighted Interval Score). The performance differences, however, seem to be partially due to a subset of time points, and the proposed model appears better probabilistically calibrated than the ensemble. We conclude that our simplified forecast model may have captured some of the dynamics of the hub ensemble, but more work is needed to understand the implicit epidemiological model that it represents.

show abstract

Section: Methodsmentioning

confidence: 99%

Evaluating an epidemiologically motivated surrogate model of a multi-model ensemble

Abbott

Sherratt

Bosse

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Whatever the method, a recognized shortcoming in the existing COVID-19 modeling literature is the lack of rigorous and robust evaluation, which is critical to assess and compare model performance. 22 On October 19th 2021, the CDC COVID-19 Forecast Hub published the EPIFORGE guidelines to attempt to improve the quality of models, highlighting the importance of consistency, interpretability, reproducibility, and comparability of models. 23 However, most model evaluation presented in the published literature remains incomprehensive.…”

Section: Introductionmentioning

confidence: 99%

“… 23 However, most model evaluation presented in the published literature remains incomprehensive. 22 Many models are evaluated for a single forecasting period, according to a single error metric, and sometimes not evaluated retrospectively at all. 22 …”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Incorporating variant frequencies data into short-term forecasting for COVID-19 cases and deaths in the USA: a deep learning approach

Du¹,

Dong²,

Badr³

et al. 2023

eBioMedicine

View full text Add to dashboard Cite

“…There have been numerous theoretical models proposed for understanding spread of infection of COVID-19 14 – 16 .…”

Section: Introductionmentioning

confidence: 99%

New compartment model for COVID-19

Odagaki

2023

Sci Rep

View full text Add to dashboard Cite

The SIR or susceptible-infected-recovered model is the standard compartment model for understanding epidemics and has been used all over the world for COVID-19. While the SIR model assumes that infected patients are identical to symptomatic and infectious patients, it is now known that in COVID-19 pre-symptomatic patients are infectious and there are significant number of asymptomatic patients who are infectious. In this paper, population is separated into five compartments for COVID-19; susceptible individuals (S), pre-symptomatic patients (P), asymptomatic patients (A), quarantined patients (Q) and recovered and/or dead patients (R). The time evolution of population in each compartment is described by a set of ordinary differential equations. Numerical solution to the set of differential equations shows that quarantining pre-symptomatic and asymptomatic patients is effective in controlling the pandemic.

show abstract

An evaluation of prospective COVID-19 modelling studies in the USA: from data to science translation

Cited by 24 publications

References 59 publications

Evaluating an epidemiologically motivated surrogate model of a multi-model ensemble

Evaluating an epidemiologically motivated surrogate model of a multi-model ensemble

Incorporating variant frequencies data into short-term forecasting for COVID-19 cases and deaths in the USA: a deep learning approach

New compartment model for COVID-19

Contact Info

Product

Resources

About