Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Abstract

1AbstractForecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways.

Aggregating Human Judgment Probabilistic Predictions of Coronavirus Disease 2019 Transmission, Burden, and Preventive Measures

“…Contrary to recent work that showed a crowd can produce more accurate forecasts for cases than deaths [ 8 ], we found that aggregate median predictions of incident deaths were more accurate than predictions of incident cases.…”

“…Human judgment has produced accurate forecasts of the progression of an infectious agent for seasonal epidemics and pandemic events [ 5–7 ]. Past work studying COVID-19 and human judgment has highlighted the potential ability of aggregate human judgment predictions to adapt to changing dynamics faster than mathematical models [ 7 , 8 ].…”

Measuring the impact of COVID-19 vaccination and immunity waning: A modelling study for Portugal

“…Nevertheless, we believe the results provide valuable highlights and show the importance of modeling in supporting decision making with the need to readjust models alongside pandemic evolution. Bosse et al suggested that the combination of mathematical modeling and human judgment can complement each other [45] . To address such challenges, a pre-existing modeling toolkit with easily extendable options is currently being developed.…”

Early human judgment forecasts of human monkeypox, May 2022

“…As demonstrated during previous infectious disease outbreaks, crowdsourced human judgment can successfully estimate many different quantities that have epidemiological meaning or are of importance to public health decision makers. [3][4][5][6][7][8] During the 2014-15 and 2015-16 influenza seasons, crowdsourced forecasts of influenza-like illness were collected and found to be among the most accurate when compared to computational forecasts. 3 Beginning on Feb 18, 2020, just before WHO declared COVID-19 a pandemic, experts in infectious disease modelling were called upon to generate predictions of a diverse set of quantities related to COVID-19, including the number of weekly incident cases at the US national level, the cumulative number of deaths by the end of 2020, and counterfactual values (eg, the reported number of cases at the state level under two different potential policy changes).…”

“…5 Additional studies have found that human judgment and computational modelling might complement one another. [6][7][8] We collected 686 unique and revised predictions (358 unique and 328 revised) from May 19 to May 24, 2022, on the Metaculus human judgment crowd sourcing platform (methods, limitations, and TRIPOD reporting guidelines are in the appendix pp 15-19).…”