Calibration of individual-based models to epidemiological data: A systematic review

Abstract

Individual-based models (IBMs) informing public health policy should be calibrated to data and provide estimates of uncertainty. Two main components of model-calibration methods are the parameter-search strategy and the goodness-of-fit (GOF) measure; many options exist for each of these. This review provides an overview of calibration methods used in IBMs modelling infectious disease spread. We identified articles on PubMed employing simulation-based methods to calibrate IBMs informing public health policy in HIV, tuberculosis, and malaria epidemiology published between 1 January 2013 and 31 December 2018. Articles were included if models stored individual-specific information, and calibration involved comparing model output to population-level targets. We extracted information on parameter-search strategies, GOF measures, and model validation. The PubMed search identified 653 candidate articles, of which 84 met the review criteria. Of the included articles, 40 (48%) combined a quantitative GOF measure with an algorithmic parameter-search strategyeither an optimisation algorithm (14/40) or a sampling algorithm (26/40). These 40 articles varied widely in their choices of parameter-search strategies and GOF measures. For the remaining 44 (52%) articles, the parameter-search strategy could either not be identified (32/44) or was described as an informal, non-reproducible method (12/44). Of these 44 articles, the majority (25/44) were unclear about the GOF measure used; of the rest, only five quantitatively evaluated GOF. Only a minority of the included articles, 14 (17%) provided a rationale for their choice of model-calibration method. Model validation was reported in 31 (37%) articles. Reporting on calibration methods is far from optimal in epidemiological modelling studies of HIV, malaria and TB transmission dynamics. The adoption of better documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology. There is a need for research comparing the performance of calibration methods to inform decisions about the parameter-search strategies and GOF measures.

COVID-19 scenarios for comparing the effectiveness of age-specific vaccination regimes, exemplified for the city of Aschaffenburg (Germany)

“…Procedures are described in the literature to allow for an automatic calibration process of agent-based models (Hazelbag et al 2020). In this case, however, manual adjustment of the model parameters proved to be the best way of fitting.…”

Inferring HIV Transmission Network Determinants Using Agent-Based Models Calibrated to Multi-Data Sources

“…We used mean relative error (MRE) to evaluate the GoF. Thus, in this study, the two main components of model-calibration were: the parameter-search strategy by LHS and the ABC, and the GoF measure by the mean relative error [14].…”

“…These types of studies use mostly epidemiological data, clinical, and sexual behavioural survey data [2,13]. Either models are fitted to these data [14,15], or statistical analysis is performed on them [3,13,15,16]. However, if we want to understand some key features of HIV infection, such as mixing between different groups of the populations of interest, the rapidity of viral spread within populations, transmission clusters and where new infections occur, and the transmission of drug resistance [17,18], we concur with the idea of undertaking phylogenetic analysis which uses viral sequence data.…”

CovidSIMVL --Transmission Trees, Superspreaders and Contact Tracing in Agent Based Models of Covid-19

“…Agent-based models -characterizing events Like other agent-based or individual-based infectious disease transmission models, 14,15,16 CovidSIMVL generates aggregate results by counting events over the course of multiple steps within a given simulation trial. These aggregate results include regularly reported metrics such at R0 at a particular simulation step.…”