Validation of population-based disease simulation models: a review of concepts and methods

Abstract

BackgroundComputer simulation models are used increasingly to support public health research and policy, but questions about their quality persist. The purpose of this article is to review the principles and methods for validation of population-based disease simulation models.MethodsWe developed a comprehensive framework for validating population-based chronic disease simulation models and used this framework in a review of published model validation guidelines. Based on the review, we formulated a set of recommendations for gathering evidence of model credibility.ResultsEvidence of model credibility derives from examining: 1) the process of model development, 2) the performance of a model, and 3) the quality of decisions based on the model. Many important issues in model validation are insufficiently addressed by current guidelines. These issues include a detailed evaluation of different data sources, graphical representation of models, computer programming, model calibration, between-model comparisons, sensitivity analysis, and predictive validity. The role of external data in model validation depends on the purpose of the model (e.g., decision analysis versus prediction). More research is needed on the methods of comparing the quality of decisions based on different models.ConclusionAs the role of simulation modeling in population health is increasing and models are becoming more complex, there is a need for further improvements in model validation methodology and common standards for evaluating model credibility.

Evidence Brief – Trends and projections of obesity among Canadians

“…Our microsimulation projections suggest that despite a stabilization in obesity prevalence among children and adolescents, if nothing new and effective is done to mitigate the trend among adults, objectively measured obesity will rise over the next two decades, affecting more than one in three Canadian adults by 2031. These projections from a validated population health microsimulation model 18 provide a realistic benchmark against which to compare new surveillance data to gauge shifts in the trend of obesity among Canadian children and adults. Objectively measured prevalence from Canadian population health surveys (see Data Sources for details).…”

UKPDS Outcomes Model 2: a new version of a model to simulate lifetime health outcomes of patients with type 2 diabetes mellitus using data from the 30 year United Kingdom Prospective Diabetes Study: UKPDS 82

“…We carried out one-way deterministic sensitivity analysis to increase understanding of the relationship between model inputs and outputs [20,22] and to determine the relative importance of patient characteristics in driving aggregate outcomes of life expectancy. Using patient-level data from the LDS as inputs, we investigated the impact on remaining life expectancy of individually changing continuous risk factors by ±1 SD of the mean and of doubling and halving the rates of binary variables such as smoking.…”

Synthèse portant sur les données probantes – Tendances et projections relatives à l’obésité chez les Canadiens

“…Nos projections laissent penser que, malgré une stabilisation de la prévalence de l'obésité chez les enfants et les adolescents, si aucune nouvelle mesure efficace n'est mise en place relativement à la tendance chez les adultes, l'obésité mesurée objectivement augmentera au cours des deux prochaines décennies, affectant plus d'1 adulte canadien sur 3 d'ici 2031. Ces projections obtenues à l'aide d'un modèle validé de microsimulation de santé de la population 18 . fournissent un point de référence réaliste pour comparer les nouvelles données de surveillance afin de mettre en lumière les changements dans les tendances relatives à l'obésité chez les enfants et les adultes canadiens.…”