Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States

Abstract

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policy makers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.

Comparison of Combination Methods to Create Calibrated Ensemble Forecasts for Seasonal Influenza in the U.S

“…As demonstrated in the forecasting literature, in many settings ensemble forecasts have consistent superior performance and give decision makers the ability to unify the strengths and diversity of individual models into one forecast. These particular advantages are of great importance in practice for infectious disease forecasting [9,12,21,[26][27][28]. This work aims to offer insight into forecast accuracy and calibration of parametric combination methods in which calibration and individual model weight estimation happen simultaneously in the application of seasonal influenza forecasting in the U.S.…”

“…More parsimonious combination methods with fixed, equally weighted individual model weights, namely the EW-LP, EW-BLP and EW-BMC 2 , appear to not be flexible enough to deliver superior performance compared to the other methods in this study. While this is the case for this application, combination methods using equal weights with or without the beta transformation could be useful in other applications where it might be difficult to estimate individual model weights when available models change over time or training data are limited [27].…”

“…A ensemble forecast unifies signals from many models into a single forecast, making it easier for stakeholders to understand. In addition, ensemble forecasts have been shown to consistently achieve a high degree of accuracy and often outperform individual forecasts of infectious disease targets [9,11,12,23,26,27,32,34,35]. A subset of teams participating in the FluSight challenge has produced a collaborative multimodel ensemble, the FluSight Network ensemble, using stacked generalization -in particular, the FluSight Network ensemble is calculated as a linear combination of the individual forecasts.…”

Using digital traces to build prospective and real-time county-level early warning systems to anticipate COVID-19 outbreaks in the United States

“…By implementing event detection algorithms on each of these internet-based time series and using machine learning strategies to combine this information, we anticipate the onset of local COVID-19 outbreaks-defined as the time when the local effective reproductive number, R t , becomes larger than 1 in a given region (51). In comparison to the current state-of-the-art models on COVID-19 prediction led by the CDC's COVID-19 Forecasting Hub Consortium (23,(52)(53)(54), our methods do not aim to predict the number of cases or deaths, but rather were designed to detect sharp increases in COVID-19 activity, a task that the current CDC models have continuously failed to accomplish as stated by Cramer et al (52): "Most forecasts [within the CDC's COVID-19 Forecast Hub Consortium] have failed to reliably predict rapid changes in the trends of reported cases and hospitalizations. Due to this limitation, they should not be relied upon for decisions about the possibility or timing of rapid changes in trends."…”