Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

Abstract

Background Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported from a standardised source over the next one to four weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance. Results Over 52 weeks we collected and combined up to 28 forecast models for 32 countries. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 84% of participating models’ forecasts of incident cases (with a total N=862), and 92% of participating models’ forecasts of deaths (N=746). Across a one to four week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over four weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than two weeks. Code and data availability All data and code are publicly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.

Learning from the past: a short term forecast method for the COVID-19 incidence curve

“…In the first week horizon, the Ensemble method is the second best, and in the second, third and fourth week the MUNI-ARIMA method, which is very close to EpiLearn , ranks second and the Ensemble method third. The result of this ensemble model is usually considered as the best option as argued in [12]. A great advantage of using the scores published by the European Hub is that such scores cannot be manipulated.…”

“…The European Covid-19 Forecast Hub [9] also proposes “an ensemble, or model average, of submitted forecasts to the European COVID-19 Forecast Hub”, described in [12]. In it, the teams submit weekly forecasts for COVID-19 cases and deaths in up to 32 countries for the next week and the three following weeks.…”

“…In [12], the authors report that the ensemble performed better on relative WIS than 84% of participating models’ forecasts of incident cases (with a total N=862), and 92% of participating models’ forecasts of deaths (N=746). Across a one to four week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over four weeks for incident death forecasts.…”

Evaluating the use of social contact data to produce age-specific forecasts of SARS-CoV-2 incidence

“…As such a host of approaches have been developed to make short term epidemiological forecasts. A popular genre of methodology for infectious disease forecasts are renewal-equation based 'semi-mechanistic' models [2,[4][5][6], which infer key epidemiological parameters from historical time-series data, in particular changes in transmission intensity, and use them to forecast future epidemic dynamics without requiring the more detailed assumptions and complex mathematical framework involved in 'fully-mechanistic' models (e.g. compartmental or agent based models).…”

A scenario modelling analysis to anticipate the impact of COVID-19 vaccination in adolescents and children on disease outcomes in the Netherlands, summer 2021

“…Scenario modelling, in an infectious disease context, aims to provide long-term projections of epidemic trajectories under different scenarios [ 4 ], and can provide useful insight about the likely direction and magnitude of change (between scenarios) and the trade-offs between different interventions [ 5 ]. Unlike, forecasting, which aims to predict what will happen in a short time frame (typically, a few weeks) [ 6 , 7 ], scenario modelling usually covers many months [ 8 ], providing bounds for outbreak trajectories. This provides policymakers with more insight and perspective to make decisions, which are usually most effective with regards to epidemics if they can be made before the modelled scenarios actually occur.…”