A Unifying Framework for Probabilistic Validation Metrics

Abstract

Probabilistic modeling methods are increasingly being employed in engineering applications. These approaches make inferences about the distribution for output quantities of interest. A challenge in applying probabilistic computer models (simulators) is validating output distributions against samples from observational data. An ideal validation metric is one that intuitively provides information on key differences between the simulator output and observational distributions, such as statistical distances/divergences. Within the literature, only a small set of statistical distances/divergences have been utilized for this task; often selected based on user experience and without reference to the wider variety available. As a result, this paper offers a unifying framework of statistical distances/divergences, categorizing those implemented within the literature, providing a greater understanding of their benefits, and offering new potential measures as validation metrics. In this paper, two families of measures for quantifying differences between distributions, that encompass the existing statistical distances/divergences within the literature, are analyzed: f-divergence and integral probability metrics (IPMs). Specific measures from these families are highlighted, providing an assessment of current and new validation metrics, with a discussion of their merits in determining simulator adequacy, offering validation metrics with greater sensitivity in quantifying differences across the range of probability mass.

Measuring data similarity in population-based structural health monitoring using distance metrics

“…M is a measurable space, f is a convex function, F is a class of real-valued bounded measurable functions on M, f is the subset of functions that define the metric, and sup is the supremum: the least upper bound of point-wise difference. 30,32 Another (true) metric that is considered here is the maximum mean discrepancy (MMD), which circumvents the need for density estimation (this is considered beneficial as there is no universally accepted methodology for conducting density estimations, and they can introduce unwanted variability into the metric formulations). The MMD uses a kernel function to obtain the maximum distance between the mean embeddings of two features or vectors that have been mapped into a reproducing kernel Hilbert space (RKHS).…”

“… M is a measurable space, ϕ is a convex function, F is a class of real-valued bounded measurable functions on ℳ , f is the subset of functions that define the metric, and sup is the supremum: the least upper bound of point-wise difference. 30,32…”

“…34,35 In the past, the MMD has been used in a variety of disciplines, because of its strength in assessing similarity of a range of data types; from time series, 36,37 graphical data, 38 images, 39 to attribute matching. 34,40 The MMD has been applied successfully in the field of verification and validation in SHM, 32 for domain adaptation, 20,41,42 in computer sciences to distinguish between malicious and honest users, 43 and to evaluate the effectiveness of treatment and diagnosis in cancer research 44 to name a few. The biased estimate of the MMD is given by,…”

On Digital Twins, Mirrors, and Virtualizations: Frameworks for Model Verification and Validation

“…If the comparison were made on the whole predictive or parameter density functions, i.e. the scenario in which the predictive distribution is compared to observational distribution (often via a finite sample set), one might define a statistical distance (or divergence) measure [15,17], for example, a Hellinger distance, leading to the definition of an α−mirror as a Hellinger-mirror etc.…”

Physics-Informed Machine Learning for Structural Health Monitoring

“…Loading is often unknown and unmeasured, and dynamic behaviour during operation needs to be fully captured by a computational model, but is sensitive to small changes in (or disturbances to) the structure. Validation and updating of large complex models bring their own challenges and remain active research areas [4][5][6][7].…”