Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics

Sarin, H.; Kokkolaras, Michael; Hulbert, Gregory M.; Papalambros, Panos Y.; Barbat, Saeed; Yang, Ren‐Jye

doi:10.1115/1.4002478

Cited by 85 publications

(49 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second is the Normalized Mean Square Error (NMSE), which measures the error between them, exposing the most striking differences. As mentioned in (Sarin et al, 2010), correlation and error metrics are the most used approach on testing simulation outcomes. Finally, the third technique is the Blend-Altman plot (Altman and Bland, 1983), a graphic metric capable of analyzing the concordance between two data sets.…”

Section: Obtained Resultsmentioning

confidence: 99%

Scorpius: sFlow Network Anomaly Simulator

Assis¹,

Proença²

2015

Journal of Computer Science

View full text Add to dashboard Cite

Due to the increasing amount of data traveling computer networks every day, efficient management of this information is required to ensure the quality of the services provided by them. Development of new network management tools and mechanisms is a widely approached area due to its importance, not only to the current technology, but also to next generation network standards and equipments. Several researches have been directed to the use of IP Flows in order to increase the efficiency of these management tools. Although there are several proposed approaches in this area, most of them don't have suitable test scenarios to validate their performance results. In this study, we present Scorpius, a new simulation tool able to help testing network management mechanisms based on IP Flows. Scorpius is capable of simulating different kinds of anomalies, such as Denial of Service (DoS), Distributed Denial of Service (DDoS), Flash Crowd and Port Scan, directly into the flow export files. This characteristic unites the advantages of tests in real network environments without the drawbacks of the occurrence of real anomalies and attacks, even controlled ones. This approach makes the processes of performance analysis of anomaly detection approaches easier, without interfering or hampering the operation of the analyzed network. In order to validate the efficiency of the presented tool, we use real data collected from a large-scale network environment.

show abstract

Section: Obtained Resultsmentioning

confidence: 99%

Scorpius: sFlow Network Anomaly Simulator

Assis¹,

Proença²

2015

Journal of Computer Science

View full text Add to dashboard Cite

show abstract

“…Quantitative comparisons consist of comparing defined error measures or error metrics (validation metrics). Reference [6] makes the following distinction between an error measure and an error metric: "An error measure provides a quantitative value associated with differences in a particular feature of time series. An error metric provides an overall quantitative value of the discrepancy between time series; it can be a single error measure or a combination of error measures".…”

Section: Verificationmentioning

confidence: 99%

“…Reference [6] proposes three error measures describing the error in magnitude, the error in phase and the error in slope by combining existing measures. The three measures are then combined into a single validation metric based on linear regression using Subject Matter Expert (SME) ratings.…”

Section: Johansen's Energymentioning

confidence: 99%

“…This makes it highly dependent on the SMEs and it will therefore not be possible to compare the quantitative results of comparisons between two different models to a single set of "true" data (or test data) made using two different sets of SMEs, unless the SMEs' assessment is the same and given that the SME exists. The error measure proposed by [6] is not used further in this paper as the metric is heavily dependent on SMEs. This causes it to lose a lot of the required objectivity of a quantitative validation metric.…”

Section: Johansen's Energymentioning

confidence: 99%

“…From the above mentioned studies ( [6], [18]) in which various error measures/metrics were evaluated, it would seem that the two most likely error measures/metrics to give the most reliable validation results are Russell's error measure and Sprague & Geers' metric. These two metrics will now be discussed in more detail.…”

Section: Johansen's Energymentioning

confidence: 99%

See 2 more Smart Citations

Validation metric based on relative error

Kat

Schalk

2012

Mathematical and Computer Modelling of Dynamical Systems

View full text Add to dashboard Cite

-Engineers and scientists are often faced with the problem of objectively comparing time histories of measured and/or simulated data. This paper presents a reliable and intuitive validation metric for use in the validation process. The proposed validation metric is able to quantify the agreement/disagreement between deterministic system response quantities of interest obtained from measurements on a physical system and predictions from a mathematical model. The validation metric is based on the relative error and the challenges concerning the use of the relative error on periodic signals are addressed. The validation metric is compared to similar metrics and their advantages and limitations are discussed. The results show that the proposed validation metric gives a comprehensive error that is able to quantify the agreement between two periodic signals and is easily interpretable.

show abstract

Coupled finite element analysis of the dynamics of poroelastic media considering the relative fluid acceleration

Xu,

Uzuoka,

Ueda

2024

Num Anal Meth Geomechanics

View full text Add to dashboard Cite

This paper mainly discusses the dynamics of poroelastic media using the finite element analysis based on the u‐v‐p full formulation, where , , and p denote the solid displacement, relative fluid velocity with respect to solid velocity, and pore fluid pressure. It incorporates the effect of relative fluid acceleration with respect to solid acceleration on soil dynamic response. The ‐‐p formulation is first verified through the comparison with the analytical solution. After that, the response of one‐dimensional saturated soil column and two‐dimensional partially saturated soil layer subjected to vertical loading with different combinations of soil permeability and loading frequency are investigated using the analyses with the ‐‐p and ‐p formulations, based on which the effect of relative pore fluid motion is discussed and the differences in the soil response between two kinds of analysis approaches are examined. Results reveal that the rapid fluid flow has a pronounced effect on soil dynamics and the soil with a greater degree of saturation is more sensitive to the increase of loading frequency and soil permeability. The soil dynamic response can basically be divided into two main categories that represent insignificant and significant flow motion depending on loading frequency and soil permeability. Furthermore, despite vertical loading, horizontal soil response can also be affected by the large relative pore fluid flow when loading frequency and soil permeability are large. In addition, the simplified formulation can still be applicable to predict dynamics of poroelastic media in cases with high loading frequency and low soil permeability.

show abstract

Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics

Cited by 85 publications

References 15 publications

Scorpius: sFlow Network Anomaly Simulator

Scorpius: sFlow Network Anomaly Simulator

Validation metric based on relative error

Coupled finite element analysis of the dynamics of poroelastic media considering the relative fluid acceleration

Contact Info

Product

Resources

About