Model uncertainty and decision making: Predicting the Impact of COVID-19 Using the CovidSim Epidemiological Code

Abstract

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV2) virus has rapidly spread worldwide since December 2019, and early modelling work of this pandemic has assisted in identifying effective government interventions. The UK government relied in part on the CovidSim model developed by the MRC Centre for Global Infectious Disease Analysis at Imperial College London, to model various non-pharmaceutical intervention strategies, and guide its government policy in seeking to deal with the rapid spread of the COVID-19 pandemic during March and April 2020. CovidSim is subject to different sources of uncertainty, namely parametric uncertainty in the inputs, model structure uncertainty (i.e. missing epidemiological processes) and scenario uncertainty, which relates to uncertainty in the set of conditions under which the model is applied. We have undertaken an extensive parametric sensitivity analysis and uncertainty quantification of the current CovidSim code. From the over 900 parameters that are provided as input to CovidSim, we identified a key subset of 20 parameters to which the code output is most sensitive. We find that the uncertainty in the code is substantial, in the sense that imperfect knowledge in these inputs will be magnified to the outputs, up to the extent of ca. 300%. Most of this uncertainty can be traced back to the sensitivity of three parameters. Compounding this, the model can display significant bias with respect to observed data, such that the output variance does not capture this validation data with high probability. We conclude that quantifying the parametric input uncertainty is not sufficient, and that the effect of model structure and scenario uncertainty cannot be ignored when validating the model in a probabilistic sense.

Uncertainty quantification in classical molecular dynamics

“…Uncertainties in a model's parameters contribute to the uncertainties in its predictions. Careful and systematic uncertainty quantification applied to the parameters enables one to estimate prediction uncertainty based on parameter uncertainty inflation or deflation [13,19]. In some instances, prediction errors due to model inadequacy can be handled by statistical correction of predictions, which may provide a reliable uncertainty measure [20].…”

Pandemic Modeling, Good and Bad

“…More generally, later work suggests that the projections of "Report 9" are highly sensitive not only to its estimates of parameter values but also to omitted factors (see the next two sections) and to uncertainty about which conditions actually apply (Edeling et al 2020;Winsberg, Brennan, and Surprenant 2021). This reinforces the point that the report's own sensitivity analyses are not enough.…”

When we can trust computers (and when we can't)

“…The VECMA toolkit is already being applied in several circumstances: climate modelling, where multiscale simulations of the atmosphere and oceans are required; forecasting refugee movements away from conflicts, or as a result of climate change, to help prioritize resources and investigate the effects of border closures and other policy decisions [49]; for exploring the mechanical properties of a simulated material at several length and time scales with verified multiscale simulations; and multiscale simulations to understand the mechanisms of heat and particle transport in fusion devices, which is important because the transport plays a key role in determining the size, shape and more detailed design and operating conditions of a future fusion power reactor, and hence the possibility of extracting almost limitless energy; and verified simulations to aid in the decision-making of drug prescriptions, simulating how drugs interact with a virtual version of a patient's proteins, [50] or how stents will behave when placed in virtual versions of arteries [51]. The toolkit has also been used to demonstrate the very considerable uncertainty in the predictions arising from the CovidSim code used to make predictions of death rates caused by the COVID-19 pandemic [52,53].…”