Results from the second year of a collaborative effort to forecast influenza seasons in the United States

Abstract

Accurate forecasts could enable more informed public health decisions. Since 2013, CDC has worked with external researchers to improve influenza forecasts by coordinating seasonal challenges for the United States and the 10 Health and Human Service Regions. Forecasted targets for the 2014-15 challenge were the onset week, peak week, and peak intensity of the season and the weekly percent of outpatient visits due to influenza-like illness (ILI) 1-4 weeks in advance. We used a logarithmic scoring rule to score the weekly forecasts, averaged the scores over an evaluation period, and then exponentiated the resulting logarithmic score. Poor forecasts had a score near 0, and perfect forecasts a score of 1. Five teams submitted forecasts from seven different models. At the national level, the team scores for onset week ranged from <0.01 to 0.41, peak week ranged from 0.08 to 0.49, and peak intensity ranged from <0.01 to 0.17. The scores for predictions of ILI 1-4 weeks in advance ranged from 0.02-0.38 and was highest 1 week ahead. Forecast skill varied by HHS region. Forecasts can predict epidemic characteristics that inform public health actions. CDC, state and local health officials, and researchers are working together to improve forecasts.

Evaluation of Nowcasting for Detecting and Predicting Local Influenza Epidemics, Sweden, 2009–2014

“…In the first Centers for Disease Control and Prevention (CDC) challenge, a prospective study of state-of-the-art methods in which 4 aspects of influenza epidemics (start week, peak week, peak percentage, and duration) were forecasted by using routine data ( 15 ), none of the evaluated methods showed satisfactory performance for all aspects. Similarly, in the second CDC challenge, in which 3 aspects of influenza epidemics (start week, peak week, and peak intensity) were forecasted by using 7 methods, none of the evaluated methods displayed satisfactory performance ( 16 ). These challenge studies have substantially helped to widen the understanding of the difficulties of forecasting different aspects of influenza epidemics.…”

“…The main strength of the study is that it prospectively evaluates an integrated influenza nowcasting method in a local community. On the basis of experiences from previous studies ( 4 , 15 , 16 , 20 ), we considered timeliness to be the most valid general evaluation metric for our purposes. To be able to accurately support adjustments of local healthcare capacity, we used daily data for the evaluations.…”

Forecasting seasonal influenza in the U.S.: A collaborative multi-year, multi-model assessment of forecast performance

“…In part due to the growing recognition of the importance of systematically integrating forecasting into public health outbreak response, large-scale collaborations have been used in forecasting applications to develop common data standards and facilitate comparisons across multiple models. [8,9,10,11] By enabling a standardized comparison in a single application, these studies greatly improve our understanding of which models perform best in certain settings, of how results can best be disseminated and used by decision-makers, and of what the bottlenecks are in terms of improving forecasts.…”

“…While multi-model comparisons exist in the literature for single-outbreak performance [8,11,10], here we compare a consistent set of models over seven influenza seasons. To our knowledge, this is the first documented comparison of multiple real-time forecasting models from different teams across multiple seasons for any infectious disease application.…”

“…FluSight) each influenza season, soliciting prospective, real-time weekly forecasts of regional-level weighted influenza-like illness (wILI) measures from teams across the world (Figure 1). [8,10] The FluSight challenge focuses on forecasts of the weighted percentage of doctor's office visits where the patient showed symptoms of an influenza-like illness in a particular region. Weighting is done by state population as the data are aggregated to the regional and national level.…”