Metamodeling for Policy Simulations with Multivariate Outcomes

Abstract

Purpose Metamodels are simplified approximations of more complex models that can be used as surrogates for the original models. Challenges in using metamodels for policy analysis arise when there are multiple correlated outputs of interest. We develop a framework for metamodeling with policy simulations to accommodate multivariate outcomes. Methods: We combine 2 algorithm adaptation methods—multitarget stacking and regression chain with maximum correlation—with different base learners including linear regression (LR), elastic net (EE) with second-order terms, Gaussian process regression (GPR), random forests (RFs), and neural networks. We optimize integrated models using variable selection and hyperparameter tuning. We compare the accuracy, efficiency, and interpretability of different approaches. As an example application, we develop metamodels to emulate a microsimulation model of testing and treatment strategies for hepatitis C in correctional settings. Results Output variables from the simulation model were correlated (average ρ = 0.58). Without multioutput algorithm adaptation methods, in-sample fit (measured by R2) ranged from 0.881 for LR to 0.987 for GPR. The multioutput algorithm adaptation method increased R2 by an average 0.002 across base learners. Variable selection and hyperparameter tuning increased R2 by 0.009. Simpler models such as LR, EE, and RF required minimal training and prediction time. LR and EE had advantages in model interpretability, and we considered methods for improving the interpretability of other models. Conclusions In our example application, the choice of base learner had the largest impact on R2; multioutput algorithm adaptation and variable selection and hyperparameter tuning had a modest impact. Although advantages and disadvantages of specific learning algorithms may vary across different modeling applications, our framework for metamodeling in policy analyses with multivariate outcomes has broad applicability to decision analysis in health and medicine.

Emulator-based Bayesian calibration of the CISNET colorectal cancer models

“…As in previous studies, our emulators proved to be an efficient alternative for conducting analysis based on complex microsimulation models and enabling computationally intensive processes such as Bayesian calibrations (11,33). The emulator-based Bayesian calibration conducted in this work provides a fast solution to calibrate multitarget models and obtain uncertainty of the model parameters.…”

“…Although there are advantages and disadvantages of specific emulator algorithms that may vary across different model applications, our framework for metamodeling using ANN can be employed in applications with a high number of inputs and outputs, which is often the case with simulation models in the field of medical decision making. Although GPs have shown highly accurate predictions in metamodeling studies using small datasets or few inputs and outputs, the time to fit the GP emulator increases exponentially as the number of parameters in the simulator that require calibration increases, eventually becoming an intractable problem (11,16,35). In these cases, neural networks are more suitable because they scale better in terms of dimensionality than GPs and other emulators (36).…”

“…One approach to reducing the computational burden uses an emulator or metamodel that serves as a surrogate of the original complex simulation model (often referred to as a simulator) by mapping the relationship between the inputs and outputs of the simulator (11,12). An emulator is a proxy model that is less complex and faster than a simulator, can replace a simulator to predict outcomes, and is often used in other model-based analyses, such as calibration procedures (8), policy optimization (11,13), cost-effectiveness analysis (14), extrapolating findings to other settings (15) and probabilistic analysis (16). Using emulators for Bayesian calibration can reduce the computational time to obtain a sample from the model parameters posterior distribution, which has already been shown to work with a considerable reduction of time (8,17).…”

The Application of Artificial Intelligence in Health Policy: A Scoping Review

“…Another study evaluated the usability of the MIDAS (Meaningful Integration of Data Analytics and Services) Project, a big data platform for health policymaking involving several international partners and pilot sites [17] and as a secure, effective, and integrated solution to deal with health data, providing support for health policy decision-making, planning of public health activities and the implementation of the Health in All Policies approach [18]. Metamodeling is an important tool for making results from complex models accessible to decision-makers to analyze policies with multivariate outcomes [19]. Following a global perspective, another study introduced the Health Information Platform that covers two dimensions: to analyze and disseminate theoretical and technological progress of the primary research in global health and to monitor real-time emergencies, epidemic situations, and hot topics related to the eld of global health.…”

When Is Mass Prophylaxis Cost-Effective for Epidemic Control? A Comparison of Decision Approaches

“…9 Previous studies used machine learning meta-models to make clinically relevant disease screening and treatment decisions. [10][11][12][13][14][15][16] A meta-model is a statistical approximation of an original model. Common methods for developing meta-models include linear and logistic regression, generalized linear and additive models, random forest and tree methods, and neural networks.…”