Methods for model selection in applied science and engineering.

Abstract

Mathematical models are developed and used to study the properties of complex systems and/or modify these systems to satisfy some performance requirements in just about every area of applied science and engineering. A particular reason for developing a model, e.g., performance assessment or design, is referred to as the model use. Our objective is the development of a methodology for selecting a model that is sufficiently accurate for an intended use.Information on the system being modeled is, in general, incomplete, so that there may be two or more models consistent with the available information. The collection of these models is called the class of candidate models. Methods are developed for selecting the optimal member from a class of candidate models for the system. The optimal model depends on the available information, the selected class of candidate models, and the model use.Classical methods for model selection, including the method of maximum likelihood and Bayesian methods, as well as a method employing a decisiontheoretic approach, are formulated to select the optimal model for numerous applications. There is no requirement that the candidate models be random. Classical methods for model selection ignore model use and require data to be available. Examples are used to show that these methods can be unreliable when data is limited. The decision-theoretic approach to model selection does not have these limitations, and model use is included through an appropriate utility function. This is especially important when modeling high risk systems, where the consequences of using an inappropriate model for the system can be disastrous. 3The decision-theoretic method for model selection is developed and applied for a series of complex and diverse applications. These include the selection of the: (1) optimal order of the polynomial chaos approximation for non-Gaussian random variables and stationary stochastic processes, (2) optimal pressure load model to be applied to a spacecraft during atmospheric re-entry, and (3) optimal design of a distributed sensor network for the purpose of vehicle tracking and identification.4

A decision-theoretic method for surrogate model selection.

“…We note that definitions of model utility are problem-dependent; alternative definitions can be used. The utility used here is consistent with the formulation for general problems in prediction discussed in [5], Section 2.3.2.3.…”

“…Consistent with the approach developed in [5,13] we propose to instead assess the utility of each candidate surrogate model for the intended model use, then select g ∈ G that has the least expected utility. We refer to this approach as the decision-theoretic method for surrogate model selection.…”

“…More sophisticated methods are available to select locations for calibration and validation data (see, for example, [18], Section 4.3); these methods can easily be included in the model selection framework presented here. A discussion on the sensitivity of the optimal model by the decisiontheoretic method to changes in the utility function are discussed in [5], Sections 2.3.1.5, 2.4, 5.6.4, and 5.8.4.…”

“…Classical methods to select a surrogate, such as the approaches discussed in [7], Chapter 2, and/or [14], typically do not consider the model use. It has been demonstrated (see [5], Section 3.1.3) that this limitation may render classical methods for surrogate model selection inappropriate for some applications. Herein, we apply a decision-theoretic method, introduced in [5,13], to select a surrogate model for the system; the proposed approach uses elements from decision theory [2], where the model use is included via an appropriate utility function.…”

“…Herein, we apply a decision-theoretic method, introduced in [5,13], to select a surrogate model for the system; the proposed approach uses elements from decision theory [2], where the model use is included via an appropriate utility function. We apply the methods for surrogate model selection to applications in design and prediction under uncertainty; the method has been developed and applied to more general classes of models [5].…”