Using Active Learning for Speeding up Calibration in Simulation Models

Abstract

Background Most cancer simulation models include unobservable parameters that determine the disease onset and tumor growth. These parameters play an important role in matching key outcomes such as cancer incidence and mortality and their values are typically estimated via lengthy calibration procedure, which involves evaluating large number of combinations of parameter values via simulation. The objective of this study is to demonstrate how machine learning approaches can be used to accelerate the calibration process by reducing the number of parameter combinations that are actually evaluated. Methods Active learning is a popular machine learning method that enables a learning algorithm such as artificial neural networks to interactively choose which parameter combinations to evaluate. We develop an active learning algorithm to expedite the calibration process. Our algorithm determines the parameter combinations that are more likely to produce desired outputs, therefore reduces the number of simulation runs performed during calibration. We demonstrate our method using previously developed University of Wisconsin Breast Cancer Simulation Model (UWBCS). Results In a recent study, calibration of the UWBCS required the evaluation of 378,000 input parameter combinations to build a race-specific model and only 69 of these combinations produced results that closely matched observed data. By using the active learning algorithm in conjunction with standard calibration methods, we identify all 69 parameter combinations by evaluating only 5620 of the 378,000 combinations. Conclusion Machine learning methods hold potential in guiding model developers in the selection of more promising parameter combinations and hence speeding up the calibration process. Applying our machine learning algorithm to one model shows that evaluating only 1.49% of all parameter combinations would be sufficient for the calibration.

Bayesian Methods for Calibrating Health Policy Models: A Tutorial

“…The worked example describes several techniques that can be employed for calibration, but does not cover the universe of algorithms and software that might be employed. Given the proliferation of Bayesian methods, this list is large and expanding, with contributions from fields such as engineering and machine learning (41, 42). For regression-type models and relatively simple mechanistic models, available software can automate many aspects of calibration (27, 28).…”

Multiobjective Calibration of Disease Simulation Models Using Gaussian Processes

“…Bayesian calibration and active learning, which have been cited as viable approaches to calibrating cancer simulation models, can contribute to calibrating cancer simulation models by updating its knowledge of the parameter space as more parameter combinations are selected. 28,29 In the future, we will also test how well the calibration performs against age-stratified outcomes. If this is unsuccessful, we will identify age groups where the simulation error cannot be ignored for each outcome.…”

High-throughput cancer hypothesis testing with an integrated PhysiCell-EMEWS workflow

“…Since their complexity limits the use of formal analytical approaches, the calibration and interpretation of complex ABMs often requires heuristic model exploration approaches that adaptively evaluate large numbers of simulations. These approaches often involve complex iterative workflows driven by sophisticated ME algorithms, such as genetic algorithms [43] or active learning [44, 45], which adaptively refine model parameters through the analysis of recently generated simulation results and launch new simulations.…”