National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

Bracher, Johannes; Wolffram, Daniel; Deuschel, Jannik; Goergen, Konstantin; Ketterer, Jakob; Ullrich, Alexander; Abbott, Sam; Barbarossa, Maria Vittoria; Bertsimas, Dimitris; Bhatia, Sangeeta; Bodych, Marcin; Bosse, Nikos I.; Burgard, Jan Pablo; Fiedler, Jochen; Fuhrmann, Jan; Funk, Sebastian; Gambin, Anna; Gogolewski, Krzysztof; Heyder, Stefan; Hotz, Thomas; Kheifetz, Yuri; Kirsten, Holger; Krueger, Tyll; Krymova, Ekaterina; Leithaeuser, Neele; Li, Michael Lingzhi; Meinke, Jan H.; Miasojedow, Błażej; Möhring, Jan; Nouvellet, Pierre; Nowosielski, Jędrzej; Ożański, Tomasz; Radwan, Maciej; Rakowski, Franciszek; Scholz, Markus; Soni, Saksham; Srivastava, Ajitesh; Gneiting, Tilmann; Schienle, Melanie

doi:10.1101/2021.11.05.21265810

Cited by 9 publications

(12 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, we refrained from adjusting model outputs or refining the models based on past performance. Forecasts were created in real time over a period of 21 weeks from October 12th 2020 until March 1st 2021 and submitted to the German and Polish Forecast hub (Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Castro, et al, 2021;Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Fiedler, et al, 2021). All code and tools necessary to generate the forecasts and make a forecast submission are available in the covid.german.forecasts R package (Bosse, Abbott, EpiForecasts, et al, 2020a).…”

Section: Methodsmentioning

confidence: 99%

“…Viboud et al, 2018), chikungunya (e.g. Del Valle et al, 2018;Farrow et al, 2017) and now COVID-19 (Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Castro, et al, 2021;Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Fiedler, et al, 2021;Cramer et al, 2021;Cramer et al, 2020;European Covid-19 Forecast Hub, 2021;e.g. Funk et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

“…Since March 2020, forecasts of COVID-19 from multiple teams have been collected, aggregated and compared by Forecast Hubs such as the US Forecast Hub (Cramer et al, 2021;Cramer et al, 2020), the German and Polish Forecast Hub (Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Castro, et al, 2021;Bracher, Wolffram, Deuschel, Görgen, Ketterer, Ullrich, Abbott, Barbarossa, Bertsimas, Bhatia, Bodych, Bosse, Burgard, Fiedler, et al, 2021) and the European Forecast Hub (European Covid-19 Forecast Hub, 2021). Often, different individual forecasts are combined 1 into a single forecast, e.g.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Bosse

Abbott

Bracher

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

1AbstractForecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Bosse

Abbott

Bracher

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…It also provides functionality to facilitate comparing forecasters even when individual forecasts are missing and offers a range of plotting functions to visualise different aspects of forecast performance. The scoringutils package is also unique in its extensive support for forecasts in a quantile format as used by various COVID-19 Forecast Hubs (Cramer et al 2021;Bracher et al 2021b;European Covid-19 Forecast Hub 2021;Bracher et al 2021c), which also make use of scoringutils for their evaluations.…”

Section: Introductionmentioning

confidence: 99%

Evaluating Forecasts with scoringutils in R

Bosse¹,

Gruson²,

Cori³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Evaluating forecasts is essential in order to understand and improve forecasting and make forecasts useful to decision-makers. Much theoretical work has been done on the development of proper scoring rules and other scoring metrics that can help evaluate forecasts. In practice, however, conducting a forecast evaluation and comparison of different forecasters remains challenging. In this paper we introduce scoringutils, an R package that aims to greatly facilitate this process. It is especially geared towards comparing multiple forecasters, regardless of how forecasts were created, and visualising results. The package is able to handle missing forecasts and is the first R package to offer extensive support for forecasts represented through predictive quantiles, a format used by several collaborative ensemble forecasting efforts. The paper gives a short introduction to forecast evaluation, discusses the metrics implemented in scoringutils and gives guidance on when they are appropriate to use, and illustrates the application of the package using example data of forecasts for COVID-19 cases and deaths submitted to the European Forecast Hub between May and September 2021.

show abstract

“…Further, our choice of baseline model affects the given performance scores in absolute terms, and more generally the choice of appropriate baseline for epidemic forecast models is not obvious when assessing infectious disease forecasts. The model used here is supported by previous work [31], yet previous evaluation in a similar context has suggested that choice of baseline affects relative performance in general [46], and future research should be done on the best choices of baseline models in the context of infectious disease epidemics.…”

Section: Discussionmentioning

confidence: 86%

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

Sherratt

Gruson

Grah

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Background Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported from a standardised source over the next one to four weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance. Results Over 52 weeks we collected and combined up to 28 forecast models for 32 countries. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 84% of participating models’ forecasts of incident cases (with a total N=862), and 92% of participating models’ forecasts of deaths (N=746). Across a one to four week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over four weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than two weeks. Code and data availability All data and code are publicly available on Github: covid19-forecast-hub-europe/euro-hub-ensemble.

show abstract

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

Cited by 9 publications

References 41 publications

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Evaluating Forecasts with scoringutils in R

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

Contact Info

Product

Resources

About