A human judgment approach to epidemiological forecasting

Abstract

Infectious diseases impose considerable burden on society, despite significant advances in technology and medicine over the past century. Advanced warning can be helpful in mitigating and preparing for an impending or ongoing epidemic. Historically, such a capability has lagged for many reasons, including in particular the uncertainty in the current state of the system and in the understanding of the processes that drive epidemic trajectories. Presently we have access to data, models, and computational resources that enable the development of epidemiological forecasting systems. Indeed, several recent challenges hosted by the U.S. government have fostered an open and collaborative environment for the development of these technologies. The primary focus of these challenges has been to develop statistical and computational methods for epidemiological forecasting, but here we consider a serious alternative based on collective human judgment. We created the web-based “Epicast” forecasting system which collects and aggregates epidemic predictions made in real-time by human participants, and with these forecasts we ask two questions: how accurate is human judgment, and how do these forecasts compare to their more computational, data-driven alternatives? To address the former, we assess by a variety of metrics how accurately humans are able to predict influenza and chikungunya trajectories. As for the latter, we show that real-time, combined human predictions of the 2014–2015 and 2015–2016 U.S. flu seasons are often more accurate than the same predictions made by several statistical systems, especially for short-term targets. We conclude that there is valuable predictive power in collective human judgment, and we discuss the benefits and drawbacks of this approach.

Forecasting the spatial transmission of influenza in the United States

“…For the United States, analyses of historical outbreaks have yielded some understanding of the spatial movement and traveling waves associated with the spread of influenza for both interpandemic (1) and pandemic seasons (2,3). However, despite the tremendous resources and efforts invested in studying the spread and prediction of influenza (1)(2)(3)(4)(5)(6)(7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19), forecast of the spatial dispersion of this pathogen in real time remains challenging.…”

Epidemic forecasts as a tool for public health: interpretation and (re)calibration

“…A critical factor in advancing these methods is to understand how epidemic forecasts are perceived and used by public health staff, and to tailor these tools to their needs 8 . Successful adoption requires forecasts to be reliable and accurate, and the expert knowledge of public health staff is an invaluable resource for meeting these criteria 9 . Successful adoption also requires public health staff to develop confidence in the forecasts, as obtained through first‐hand experience.…”

A Collaborative Multi-Model Ensemble for Real-Time Influenza Season Forecasting in the U.S

“…The top scoring model was an ensemble of human judgment forecasts. [37] By working across disciplines and research groups and by incorporating experts from government, academia and industry, this collaborative effort showed success in bringing measurable improvements in forecast accuracy and reductions in variability. We therefore are moving substantially closer to forecasts that can and should be used to complement routine, ongoing public health surveillance of infectious diseases.…”