Economic Analysis of Model Validation for a Challenge Problem

Abstract

It is now commonplace for engineers to build mathematical models of the systems they are designing, building, or testing. And, it is nearly universally accepted that phenomenological models of physical systems must be validated prior to use for prediction in consequential scenarios. Yet, there are certain situations in which testing only or no testing and no modeling may be economically viable alternatives to modeling and its associated testing. This paper develops an economic framework within which benefit–cost can be evaluated for modeling and model validation relative to other options. The development is presented in terms of a challenge problem. We provide a numerical example that quantifies when modeling, calibration, and validation yield higher benefit–cost than a testing only or no modeling and no testing option.

Validation of a hybrid electromagnetic–piezoelectric vibration energy harvester

“…In our previous work [7], a Kolmogorov-Smirnov test, as recommended for model validation of stochastic systems [21], was used at each point in the parameter space in order to validate the distribution of results. In this case, however, three output variables must be considered simultaneously for each point and the Kolmogorov-Smirnov test cannot be applied directly for multivariate data analysis.…”

Introduction: The 2014 Sandia Verification and Validation Challenge Workshop

“…These two papers are followed by five contributions that describe the participants' responses to the challenge problem [15][16][17][18][19]. Two discussion papers are also included, in which the authors do not solve the problem but instead address relevant V&V topics while using the challenge problem as context [20,21]. The special edition concludes with a summary paper that describes the responses and lessons learned [22].…”

Models, Uncertainty, and the Sandia V&V Challenge Problem

“…It is certainly the case that all of the papers presented in VV&UQ, Volume 1, fail even to acknowledge the need for such a rigorously derived decision framework. Yet, the need for such a framework should be intuitively obvious: if one changes the definition of a word such as "probability," as has been done in the papers of this journal [15][16][17][18][19][20], one should no longer use the word as though it remains as previously defined.…”