A pattern-based validation method for the credibility evaluation of simulation models

Abstract

Measuring the credibility of a simulation model has always been challenging due to its growing uncertainty and complexity. During the past decades, plenty of metrics and evaluation procedures have been developed for evaluating different sorts of simulation models. Most of the existing research focuses on the direct comparison of numerical results with a group of reference data. However, it is sometimes unsuitable for evolving dynamic models such as the multi-agent models. With the same condition, both the practical system and the simulation model perform highly dynamic actions. The credibility of the model with insufficient information, non-stationary states and changing environment is unable to acquire through a direct pair comparison. This paper presents a pattern-based validation method to complementarily extract hidden patterns that exist in both a simulation model and its reference data, and assess the model performance in a different aspect. Firstly, multi-dimensional perceptually important points strategy is modified to find the patterns exist in time-serial data. Afterward, a pattern organizing topology is applied to automatically depict required pattern from reference data and assess the performance of the corresponding simulation model. The extensive case study on three simulation models shows the effectiveness of the proposed method.

An evidence theory-based validation method for models with multivariate outputs and uncertainty

“…Currently, verification, validation, and uncertainty quantification (VV&UQ) is increasingly significant in research in complex systems modeling and simulation. 7,8 Uncertainty in modeling could be classified into aleatory uncertainty, considered as some form of inherent or stochastic uncertainty, and epistemic uncertainty, considered either as lack of or incomplete knowledge of the system modeled, are becoming clear distinctions in the risk assessment community. 9 Also the input uncertainty in modeling and simulation refers to the unknown difference between the input models used in the simulation and those that best characterize the real-world system.…”