A Frequentist Approach to Computer Model Calibration

Wong, Raymond K. W.; Storlie, Curtis B.; Lee, Thomas C.

doi:10.1111/rssb.12182

Cited by 82 publications

(102 citation statements)

References 34 publications

Supporting

Mentioning

102

Contrasting

Order By: Relevance

“…In contrast to ad hoc, frequentist, or otherwise non-Bayesian calibration frameworks (1–4), the theory underlying Bayesian approaches provides an axiomatic basis for deciding how to quantify evidence, avoiding arbitrary decisions about the relative weight to be placed on different data sources or the use of heuristics to select well-fitting parameter sets. In contexts where priors, likelihood, and model are all correctly specified, the Bayesian approach can provide a theoretically optimal summary of the evidence (5, 6), allowing a decision-maker to maximize expected utility when paired with a utility function representing his/her preferences for different outcomes (7).…”

Section: Theoretical Frameworkmentioning

confidence: 99%

Bayesian Methods for Calibrating Health Policy Models: A Tutorial

Menzies

Soeteman²,

Pandya

et al. 2017

PharmacoEconomics

View full text Add to dashboard Cite

Mathematical simulation models are commonly used to inform health policy decisions. These health policy models represent the social and biological mechanisms that determine health and economic outcomes, combine multiple sources of evidence about how policy alternatives will impact those outcomes, and synthesize outcomes into summary measures salient for the policy decision. Calibrating these health policy models to fit empirical data can provide face validity and improve the quality of model predictions. Bayesian methods provide powerful tools for model calibration. These methods summarize information relevant to a particular policy decision into (i) prior distributions for model parameters, (ii) structural assumptions of the model, and (iii) a likelihood function created from the calibration data, combining these different sources of evidence via Bayes’ theorem. This paper provides a tutorial on Bayesian approaches for model calibration, describing the theoretical basis for Bayesian calibration approaches as well as pragmatic considerations that arise in the tasks of creating calibration targets, estimating the posterior distribution, and obtaining results to inform the policy decision. These considerations, as well as the specific steps for implementing the calibration, are described in the context of an extended worked example about the policy choice to provide (or not provide) treatment for a hypothetical infectious disease. Given the many simplifications and subjective decisions required to create prior distributions, model structure, and likelihood, calibration should be considered an exercise in creating a reasonable model that produces valid evidence for policy, rather than as a technique for identifying a unique, theoretically optimal summary of the evidence.

show abstract

Section: Theoretical Frameworkmentioning

confidence: 99%

Bayesian Methods for Calibrating Health Policy Models: A Tutorial

Menzies

Soeteman²,

Pandya

et al. 2017

PharmacoEconomics

View full text Add to dashboard Cite

show abstract

“…These conditions are considerably looser than those considered in Tuo and Wu () and Wong et al . (), but consistency is a weaker claim than the efficiency claims in those works.…”

Section: Consistencymentioning

confidence: 83%

“…(), Tuo and Wu (), Plumlee () and Wong et al . () of

l false(g, h false) = \int_{double-struckX} {g false(x false) - h false(x false)}^{2} d μ false(x false),

where μ is some measure over

double-struckX

. This is a strictly proper scoring metric that weights the relative inputs by some measure μ .…”

Section: Setting and Notationmentioning

confidence: 99%

See 1 more Smart Citation

Computer Model Calibration with Confidence and Consistency

Plumlee

2019

Journal of the Royal Statistical Society Series B: Statistical Methodology

View full text Add to dashboard Cite

Summary The paper proposes and examines a calibration method for inexact models. The method produces a confidence set on the parameters that includes the best parameter with a desired probability under any sample size. Additionally, this confidence set is shown to be consistent in that it excludes suboptimal parameters in large sample environments. The method works and the results hold with few assumptions; the ideas are maintained even with discrete input spaces or parameter spaces. Computation of the confidence sets and approximate confidence sets is discussed. The performance is illustrated in a simulation example as well as two real data examples.

show abstract

“…As such, we assume some level of familiarity with tools such as noise modeling, parametric bootstrap analyses, and synthetic dataset generation. [20][21][22][23][24] In a few instances we review key ideas, but interested readers should consult the indicated references for detailed information.…”

mentioning

confidence: 99%

The role of data analysis in uncertainty quantification: case studies for materials modeling

Patrone

Kearsley

Dienstfrey

2018

2018 AIAA Non-Deterministic Approaches Conference

View full text Add to dashboard Cite

In computational materials science, mechanical properties are typically extracted from simulations by means of analysis routines that seek to mimic their experimental counterparts. However, simulated data often exhibit uncertainties that can propagate into final predictions in unexpected ways.Thus, modelers require data analysis tools that (i) address the problems posed by simulated data, and (ii) facilitate uncertainty quantification. In this manuscript, we discuss three case studies in materials modeling where careful data analysis can be leveraged to address specific instances of these issues. As a unifying theme, we highlight the idea that attention to physical and mathematical constraints surrounding the generation of computational data can significantly enhance its analysis. I ncreasingly, the computational materials science community is embracing uncertainty quantification (UQ) as a necessary component of any modeling workflow that aims to provide actionable information for industry.1-4 In many instances, this change has been driven by the observation that computational predictions, while less expensive than their experimental counterparts, are affected by uncertainties not typically encountered in laboratory settings. As a result, there has been significant work focused on identifying and quantifying uncertainties associated with simulation tools per se. 5-7 Despite this, however, much less effort has been devoted to understanding the impact of these uncertainties on subsequent analyses and usage models. Critically, such issues must be addressed if the benefits of computational approaches are to be fully realized. Within this greater context, data analysis remains a fundamental and sometimes overlooked task that can contribute to uncertainties in ways that are difficult to quantify, if

show abstract

A Frequentist Approach to Computer Model Calibration

Cited by 82 publications

References 34 publications

Bayesian Methods for Calibrating Health Policy Models: A Tutorial

Bayesian Methods for Calibrating Health Policy Models: A Tutorial

Computer Model Calibration with Confidence and Consistency

The role of data analysis in uncertainty quantification: case studies for materials modeling

Contact Info

Product

Resources

About