Statistical Validation of Engineering and Scientific Models: A Maximum Likelihood Based Metric

Abstract

Two major issues associated with model validation are addressed here. First, we present a maximum likelihood approach to define and evaluate a model validation metric. The advantage of this approach is it is more easily applied to nonlinear problems than the methods presented earlier by Trucano (1999, 2001); the method is based on optimization for which software packages are readily available; and the method can more easily be extended to handle measurement uncertainty and prediction uncertainty with different probability structures. Several examples are presented utilizing this metric. We show conditions under which this approach reduces to the approach developed previously by Hills and Trucano (2001).Secondly, we expand our earlier discussions Trucano, 1999, 2001) on the impact of multivariate correlation and the effect of this on model validation metrics. We show that ignoring correlation in multivariate data can lead to misleading results, such as rejecting a good model when sufficient evidence to do so is not available.

Statistical Validation of Engineering and Scientific Models: Validation Experiments to Application

“…First, we extend the application-based, model validation metric presented in Hills and Trucano (2001) to the Maximum Likelihood approach introduced in Hills and Trucano (2002). This method allows us to use the target application of the code to weigh the measurements made from a validation experiment so that those measurements that are most important for the application are more heavily weighted.…”

Statistical validation of engineering and scientific models : bounds, calibration, and extrapolation.

“…The third report (Hills and Trucano, 2002) focused on the application of the Maximum Likelihood method to the non-application based validation metrics developed in the first two reports. The use of Maximum Likelihood allows highly nonlinear problems with non-normally distributed uncertainties in the measurements and the model parameters to be more easily handled.…”

“…Hills and Trucano (2001) illustrate methodology based on Monte Carlo analysis to handle non-normal distributions. Hills and Trucano (2002) provide an alternative metric based on maximum likelihood, which does not require a sensitivity analysis or as many function evaluations as a Monte Carlo analysis for nonlinear, nonnormally distributed systems.…”

“…If we must calibrate, then we must have other independent knowledge to validate against. An example of a method that uses calibration with independent data for validation, is the Maximum Likelihood method discussed by Hills and Trucano (2002). In this case, calibration was used to estimate likely values of the parameters; and the significance of the estimated parameters, based on prior independent knowledge of the uncertainty distribution of the parameters, was used to evaluate validity.…”

“…We assumed that the prediction intervals were symmetric about the estimated parameters rather than the true means for the parameters. If we have prior knowledge of the parameter distributions, this assumption can be relaxed by using a maximum likelihood approach, such as was discussed by Hills and Trucano (2002).…”