Using Metamodeling to Identify the Optimal Strategy for Colorectal Cancer Screening

Koffijberg, Hendrik; Degeling, Koen; IJzerman, Maarten Joost; Coupé, Veerle M.H.; Greuter, Marjolein J.E.

doi:10.1016/j.jval.2020.08.2099

Cited by 5 publications

(11 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most simulation models in public health generate multiple outputs of interest (e.g., costs and multiple health outcomes, possibly in different population groups). 1,21,23 Because metamodeling aims to build a replacement model to link the original simulation inputs and outputs, approaches can be generalized from multioutput regression. There are 2 main approaches: problem transformation and algorithm adaptation.…”

Section: Multioutput Regressionmentioning

confidence: 99%

“…A limitation of existing methods for metamodeling is that such methods typically do not consider the challenges that arise when multiple output variables are relevant to a particular policy analysis and when those outputs are correlated (e.g., the costs and health effects of different policy choices). 1,21,23 When outputs are correlated, an approach that does not account explicitly for these correlations could lead to invalid conclusions, for example, when comparisons between the outcomes are salient (as in the ranking of alternatives on a particular measure) or when the key quantity of interest depends on multiple outcomes.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Metamodeling for Policy Simulations with Multivariate Outcomes

et al. 2022

View full text Add to dashboard Cite

Purpose Metamodels are simplified approximations of more complex models that can be used as surrogates for the original models. Challenges in using metamodels for policy analysis arise when there are multiple correlated outputs of interest. We develop a framework for metamodeling with policy simulations to accommodate multivariate outcomes. Methods: We combine 2 algorithm adaptation methods—multitarget stacking and regression chain with maximum correlation—with different base learners including linear regression (LR), elastic net (EE) with second-order terms, Gaussian process regression (GPR), random forests (RFs), and neural networks. We optimize integrated models using variable selection and hyperparameter tuning. We compare the accuracy, efficiency, and interpretability of different approaches. As an example application, we develop metamodels to emulate a microsimulation model of testing and treatment strategies for hepatitis C in correctional settings. Results Output variables from the simulation model were correlated (average ρ = 0.58). Without multioutput algorithm adaptation methods, in-sample fit (measured by R2) ranged from 0.881 for LR to 0.987 for GPR. The multioutput algorithm adaptation method increased R2 by an average 0.002 across base learners. Variable selection and hyperparameter tuning increased R2 by 0.009. Simpler models such as LR, EE, and RF required minimal training and prediction time. LR and EE had advantages in model interpretability, and we considered methods for improving the interpretability of other models. Conclusions In our example application, the choice of base learner had the largest impact on R2; multioutput algorithm adaptation and variable selection and hyperparameter tuning had a modest impact. Although advantages and disadvantages of specific learning algorithms may vary across different modeling applications, our framework for metamodeling in policy analyses with multivariate outcomes has broad applicability to decision analysis in health and medicine.

show abstract

Section: Multioutput Regressionmentioning

confidence: 99%

mentioning

confidence: 99%

Metamodeling for Policy Simulations with Multivariate Outcomes

et al. 2022

View full text Add to dashboard Cite

show abstract

“…9 Previous studies used machine learning meta-models to make clinically relevant disease screening and treatment decisions. [10][11][12][13][14][15][16] A meta-model is a statistical approximation of an original model. Common methods for developing meta-models include linear and logistic regression, generalized linear and additive models, random forest and tree methods, and neural networks.…”

mentioning

confidence: 99%

When Is Mass Prophylaxis Cost-Effective for Epidemic Control? A Comparison of Decision Approaches

Malloy

Brandeau

2022

Med Decis Making

View full text Add to dashboard Cite

Background For certain communicable disease outbreaks, mass prophylaxis of uninfected individuals can curtail new infections. When an outbreak emerges, decision makers could benefit from methods to quickly determine whether mass prophylaxis is cost-effective. We consider 2 approaches: a simple decision model and machine learning meta-models. The motivating example is plague in Madagascar. Methods We use a susceptible-exposed-infectious-removed (SEIR) epidemic model to derive a decision rule based on the fraction of the population infected, effective reproduction ratio, infection fatality rate, quality-adjusted life-year loss associated with death, prophylaxis effectiveness and cost, time horizon, and willingness-to-pay threshold. We also develop machine learning meta-models of a detailed model of plague in Madagascar using logistic regression, random forest, and neural network models. In numerical experiments, we compare results using the decision rule and the meta-models to results obtained using the simulation model. We vary the initial fraction of the population infected, the effective reproduction ratio, the intervention start date and duration, and the cost of prophylaxis. Limitations We assume homogeneous mixing and no negative side effects due to antibiotic prophylaxis. Results The simple decision rule matched the SEIR model outcome in 85.4% of scenarios. Using data for a 2017 plague outbreak in Madagascar, the decision rule correctly indicated that mass prophylaxis was not cost-effective. The meta-models were significantly more accurate, with an accuracy of 92.8% for logistic regression, 95.8% for the neural network model, and 96.9% for the random forest model. Conclusions A simple decision rule using minimal information about an outbreak can accurately evaluate the cost-effectiveness of mass prophylaxis for outbreak mitigation. Meta-models of a complex disease simulation can achieve higher accuracy but with greater computational and data requirements and less interpretability. Highlights We use a susceptible-exposed-infectious-removed model and net monetary benefit to derive a simple decision rule to evaluate the cost-effectiveness of mass prophylaxis. We use the example of plague in Madagascar to compare the performance of the analytically derived decision rule to that of machine learning meta-models trained on a stochastic dynamic transmission model. We assess the accuracy of each approach for different combinations of disease dynamics and intervention scenarios. The machine learning meta-models are more accurate predictors of mass prophylaxis cost-effectiveness. However, the simple decision rule is also accurate and may be a preferred substitute in low-resource settings.

show abstract

“…[18][19][20] In health policy, meta-models have primarily been used to lower computation time of microsimulation and other complex models, to streamline value-of-information analysis, and to assess decision uncertainty due to uncertainty in model inputs. 16,[21][22][23][24][25][26][27][28][29][30][31] Recent studies have described the use of meta-models to simplify microsimulation models of strategies for infant HIV testing and screening, 32 colorectal cancer screening, 33 and hepatitis C testing and treatment in correctional settings. 34 In this article, we propose a general, disease-and model-agnostic method that uses machine-learning methods to develop meta-models that simplify complex models for personalization of medicine treatment decisions.…”

mentioning

confidence: 99%

“…18–20 In health policy, meta-models have primarily been used to lower computation time of microsimulation and other complex models, to streamline value-of-information analysis, and to assess decision uncertainty due to uncertainty in model inputs. 16,21–31 Recent studies have described the use of meta-models to simplify microsimulation models of strategies for infant HIV testing and screening, 32 colorectal cancer screening, 33 and hepatitis C testing and treatment in correctional settings. 34…”

mentioning

confidence: 99%

Personalization of Medical Treatment Decisions: Simplifying Complex Models while Maintaining Patient Health Outcomes

Weyant

Brandeau

2021

Med Decis Making

View full text Add to dashboard Cite

Background Personalizing medical treatments based on patient-specific risks and preferences can improve patient health. However, models to support personalized treatment decisions are often complex and difficult to interpret, limiting their clinical application. Methods We present a new method, using machine learning to create meta-models, for simplifying complex models for personalizing medical treatment decisions. We consider simple interpretable models, interpretable ensemble models, and noninterpretable ensemble models. We use variable selection with a penalty for patient-specific risks and/or preferences that are difficult, risky, or costly to obtain. We interpret the meta-models to the extent permitted by their model architectures. We illustrate our method by applying it to simplify a previously developed model for personalized selection of antipsychotic drugs for patients with schizophrenia. Results The best simplified interpretable, interpretable ensemble, and noninterpretable ensemble models contained at most half the number of patient-specific risks and preferences compared with the original model. The simplified models achieved 60.5% (95% credible interval [crI]: 55.2–65.4), 60.8% (95% crI: 55.5–65.7), and 83.8% (95% crI: 80.8–86.6), respectively, of the net health benefit of the original model (quality-adjusted life-years gained). Important variables in all models were similar and made intuitive sense. Computation time for the meta-models was orders of magnitude less than for the original model. Limitations The simplified models share the limitations of the original model (e.g., potential biases). Conclusions Our meta-modeling method is disease- and model- agnostic and can be used to simplify complex models for personalization, allowing for variable selection in addition to improved model interpretability and computational performance. Simplified models may be more likely to be adopted in clinical settings and can help improve equity in patient outcomes.

show abstract

Using Metamodeling to Identify the Optimal Strategy for Colorectal Cancer Screening

Cited by 5 publications

References 38 publications

Metamodeling for Policy Simulations with Multivariate Outcomes

Metamodeling for Policy Simulations with Multivariate Outcomes

When Is Mass Prophylaxis Cost-Effective for Epidemic Control? A Comparison of Decision Approaches

Personalization of Medical Treatment Decisions: Simplifying Complex Models while Maintaining Patient Health Outcomes

Contact Info

Product

Resources

About