Nonidentifiability in Model Calibration and Implications for Medical Decision Making

Alarid‐Escudero, Fernando; MacLehose, Richard F.; Peralta, Yadira; Kuntz, Karen M.; Enns, Eva A.

doi:10.1177/0272989x18792283

Cited by 23 publications

(45 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Generally, underidentifiability would be addressed through the inclusion of additional targets (if available), narrower prior distribution ranges (if these are known), or fewer parameters (e.g., less age stratification), not calibration itself. 42…”

Section: Discussionmentioning

confidence: 99%

“…We found that the best-fitting parameter set from the initial random sampling stage was different each time we reran IMIS because of the low sampling density relative to the high dimensionality or the parameter space; 12 targets to inform 24 parameters yields underidentifiability and the possibility of many parameter sets fitting the targets equally well. 42…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Methods for Model Calibration under High Uncertainty: Modeling Cholera in Bangladesh

et al. 2020

View full text Add to dashboard Cite

Background. Published data on a disease do not always correspond directly to the parameters needed to simulate natural history. Several calibration methods have been applied to computer-based disease models to extract needed parameters that make a model’s output consistent with available data. Objective. To assess 3 calibration methods and evaluate their performance in a real-world application. Methods. We calibrated a model of cholera natural history in Bangladesh, where a lack of active surveillance biases available data. We built a cohort state-transition cholera natural history model that includes case hospitalization to reflect the passive surveillance data-generating process. We applied 3 calibration techniques: incremental mixture importance sampling, sampling importance resampling, and random search with rejection sampling. We adapted these techniques to the context of wide prior uncertainty and many degrees of freedom. We evaluated the resulting posterior parameter distributions using a range of metrics and compared predicted cholera burden estimates. Results. All 3 calibration techniques produced posterior distributions with a higher likelihood and better fit to calibration targets as compared with prior distributions. Incremental mixture importance sampling resulted in the highest likelihood and largest number of unique parameter sets to better inform joint parameter uncertainty. Compared with naïve uncalibrated parameter sets, calibrated models of cholera in Bangladesh project substantially more cases, many of which are not detected by passive surveillance, and fewer deaths. Limitations. Calibration cannot completely overcome poor data quality, which can leave some parameters less well informed than others. Calibration techniques may perform differently under different circumstances. Conclusions. Incremental mixture importance sampling, when adapted to the context of high uncertainty, performs well. By accounting for biases in data, calibration can improve model projections of disease burden.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Methods for Model Calibration under High Uncertainty: Modeling Cholera in Bangladesh

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The reasons cited for nonidentifiability include an insufficient number (or type) of calibration targets, an excessive number of unknown model parameters, a large parameter space, or an inappropriate GOF criterion. 25 We have attempted to mitigate nonidentifiability in this work by presenting a multiobjective approach that eliminates the possibility of a single GOF criterion and a conservative parameter range Θ . In addition, we note that heuristics proposed to assess nonidentifiability, such as the collinearity index, are available.…”

Section: Discussionmentioning

confidence: 99%

“…In addition, we note that heuristics proposed to assess nonidentifiability, such as the collinearity index, are available. 25…”

Section: Discussionmentioning

confidence: 99%

Multiobjective Calibration of Disease Simulation Models Using Gaussian Processes

et al. 2019

View full text Add to dashboard Cite

Background. Developing efficient procedures of model calibration, which entails matching model predictions to observed outcomes, has gained increasing attention. With faithful but complex simulation models established for cancer diseases, key parameters of cancer natural history can be investigated for possible fits, which can subsequently inform optimal prevention and treatment strategies. When multiple calibration targets exist, one approach to identifying optimal parameters relies on the Pareto frontier. However, computational burdens associated with higher-dimensional parameter spaces require a metamodeling approach. The goal of this work is to explore multiobjective calibration using Gaussian process regression (GPR) with an eye toward how multiple goodness-of-fit (GOF) criteria identify Pareto-optimal parameters. Methods. We applied GPR, a metamodeling technique, to estimate colorectal cancer (CRC)–related prevalence rates simulated from a microsimulation model of CRC natural history, known as the Colon Modeling Open Source Tool (CMOST). We embedded GPR metamodels within a Pareto optimization framework to identify best-fitting parameters for age-, adenoma-, and adenoma staging–dependent transition probabilities and risk factors. The Pareto frontier approach is demonstrated using genetic algorithms with both sum-of-squared errors (SSEs) and Poisson deviance GOF criteria. Results. The GPR metamodel is able to approximate CMOST outputs accurately on 2 separate parameter sets. Both GOF criteria are able to identify different best-fitting parameter sets on the Pareto frontier. The SSE criterion emphasizes the importance of age-specific adenoma progression parameters, while the Poisson criterion prioritizes adenoma-specific progression parameters. Conclusion. Different GOF criteria assert different components of the CRC natural history. The combination of multiobjective optimization and nonparametric regression, along with diverse GOF criteria, can advance the calibration process by identifying optimal regions of the underlying parameter landscape.

show abstract

“…Some parameters of the mathematical model, like the per-partnership transmission probability and the duration of natural immunity, cannot be informed directly by observed data, so we estimated the parameters by calibrating the transmission models (empirical and A-P) to age-specific pre-vaccination prevalence of HPV16 in Britain [29]. To account for potential non-identifiability, we adopted a Bayesian framework [30]. Briefly, we defined prior distributions on parameters of interest, then used incremental mixture importance sampling (IMIS) [31,32] to estimate posterior distributions of the parameters conditional on the prevalence data.…”

Section: Methodsmentioning

confidence: 99%

Revisiting assumptions about age-based mixing representations in mathematical models of sexually transmitted infections

Easterly

Alarid‐Escudero

Enns

et al. 2018

Vaccine

View full text Add to dashboard Cite

Background Sexual mixing between heterogeneous population subgroups is an integral component of mathematical models of sexually transmitted infections (STIs). This study compares the fit of different mixing representations to survey data and the impact of different mixing assumptions on the predicted benefits of hypothetical human papillomavirus (HPV) vaccine strategies. Methods We compared novel empirical (data-driven) age mixing structures with the more commonly-used assortative-proportionate (A-P) mixing structure. The A-P mixing structure assumes that a proportion of sexual contacts – known as the assortativity constant, typically estimated from survey data or calibrated – occur exclusively within one’s own age group and the remainder mixes proportionately among all age groups. The empirical age mixing structure was estimated from the National Survey on Sexual Attitudes and Lifestyles 3 (Natsal-3) using regression methods, and the assortativity constant was estimated from Natsal-3 as well. Using a simplified HPV transmission model under each mixing assumption, we calibrated the model to British HPV16 prevalence data, then estimated the reduction in steady-state prevalence and the number of infections averted due to expanding HPV vaccination from 12-through-26-year-old females alone to 12-year-old males or 27- to 39-year-old females. Results Empirical mixing provided a better fit to the Natsal-3 data than the best-fitting A-P structure. Using the model with empirical mixing as a reference, the model using the A-P structure often under- or overestimated the benefits of vaccination, in one case overestimating by 2-fold the number of infections prevented due to extended female catch-up in a high vaccine uptake setting. Conclusions An empirical mixing structure more accurately represents sexual mixing survey data, and using the less accurate, yet commonly-used A-P structure has a notable effect on estimates of HPV vaccination benefits. This underscores the need for mixing structures that are less dependent on unverified assumptions and are directly informed by sexual behavior data.

show abstract

Nonidentifiability in Model Calibration and Implications for Medical Decision Making

Cited by 23 publications

References 65 publications

Methods for Model Calibration under High Uncertainty: Modeling Cholera in Bangladesh

Methods for Model Calibration under High Uncertainty: Modeling Cholera in Bangladesh

Multiobjective Calibration of Disease Simulation Models Using Gaussian Processes

Revisiting assumptions about age-based mixing representations in mathematical models of sexually transmitted infections

Contact Info

Product

Resources

About