Assessing the utility of COVID-19 case reports as a leading indicator for hospitalization forecasting in the United States

Abstract

Identifying data streams that can consistently improve the accuracy of epidemiological forecasting models is challenging. Using models designed to predict daily state-level hospital admissions due to COVID-19 in California and Massachusetts, we investigated whether incorporating COVID-19 case data systematically improved forecast accuracy. Additionally, we considered whether using case data aggregated by date of test or by date of report from a surveillance system made a difference to the forecast accuracy. Evaluating forecast accuracy in a test period, after first having selected the best-performing methods in a validation period, we found that overall the difference in accuracy between approaches was small, especially at forecast horizons of less than two weeks. However, forecasts from models using cases aggregated by test date showed lower accuracy at longer horizons and at key moments in the pandemic, such as the peak of the Omicron wave in January 2022. Overall, these results highlight the challenge of finding a modeling approach that can generate accurate forecasts of outbreak trends both during periods of relative stability and during periods that show rapid growth or decay of transmission rates. While COVID-19 case counts seem to be a natural choice to help predict COVID-19 hospitalizations, in practice any benefits we observed were small and inconsistent.

Identifying real-time surveillance indicators that can be used to estimate COVID-19 hospital admissions

“…We conducted two preliminary analyses. First, we estimated the lags between each surveillance indicator and 7-day average COVID-19 hospital admissions using cross-correlation methods 7,12 . Each surveillance measure captures a different state in the disease process and is subject to different reporting efficiencies; as such, some surveillance measures are leading indicators of COVID-19 hospital admissions and others are synchronous or lagging indicators.…”

COVID-19 trends across borders: Identifying correlations among countries

“…17 Incorporating COVID-19 case data in forecasting models has shown small and inconsistent improvements in accuracy, particularly at longer horizons and during key moments in the pandemic. 18 Combining probabilistic forecasts from multiple teams using weighted combined methods has effectively generated accurate interval forecasts and predictions of probability distributions for COVID-19 mortality. 19…”

When are predictions useful? A new method for evaluating epidemic forecasts

“…In this article, we use these ensemble forecasts as test cases for our new metric: the Weighted Contextual Interval Score (WCIS). While the WCIS could easily be applied to other types of forecasting, it was designed with efforts like the COVID-19 Forecast Hub in mind.As a collaboration between modelers, public health practitioners, and government officials, the Hub is representative of efforts that will remain vital given the danger posed by both extant and heretofore unknown epidemic threats [10].However, using forecasts for consistent real time decision making remains a challenge, a vital component which is translating forecast data into actionable insights [7,11,12,13,14]. In this light, we present the WCIS as a way to alleviate challenges in this space that arise from comparing, aggregating, and interpreting forecasts made across highly spatially and temporally variable prediction instances.…”

“…However, using forecasts for consistent real time decision making remains a challenge, a vital component which is translating forecast data into actionable insights [7,11,12,13,14]. In this light, we present the WCIS as a way to alleviate challenges in this space that arise from comparing, aggregating, and interpreting forecasts made across highly spatially and temporally variable prediction instances.…”