Increasing Validity of Simulation Models Through Metamorphic Testing

Olsen, Megan; Raunak, Mohammad

doi:10.1109/tr.2018.2850315

Cited by 19 publications

(6 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work [24], the authors compared four search engines with respect to different scenarios and factors, thereby providing users and developers with a more comprehensive understanding of how to choose a proper search engine for better searching services with clear and definite objectives. Olsen and Raunak [25] applied MT for simulation validation, involving two prevalent simulation approaches: agentbased simulation and discret-event simulation. Guidelines were also provided for identifying MRs for both simulation approaches.…”

Section: Metamorphic Testing (Mt)mentioning

confidence: 99%

See 1 more Smart Citation

METTLE: A METamorphic Testing Approach to Assessing and Validating Unsupervised Machine Learning Systems

et al. 2020

View full text Add to dashboard Cite

Unsupervised machine learning is the training of an artificial intelligence system using information that is neither classified nor labeled, with a view to modeling the underlying structure or distribution in a dataset. Since unsupervised machine learning systems are widely used in many real-world applications, assessing the appropriateness of these systems and validating their implementations with respect to individual users' requirements and specific application scenarios / contexts are indisputably two important tasks. Such assessment and validation tasks, however, are fairly challenging due to the absence of a priori knowledge of the data. In view of this challenge, we develop a METamorphic Testing approach to assessing and validating unsupervised machine LEarning systems, abbreviated as METTLE. Our approach provides a new way to unveil the (possibly latent) characteristics of various machine learning systems, by explicitly considering the specific expectations and requirements of these systems from individual users' perspectives. To support METTLE, we have further formulated 11 generic metamorphic relations (MRs), covering users' generally expected characteristics that should be possessed by machine learning systems. To demonstrate the viability and effectiveness of METTLE, we have performed an experiment involving six commonly used clustering systems. Our experiment has shown that, guided by user-defined MR-based adequacy criteria, end users are able to assess, validate, and select appropriate clustering systems in accordance with their own specific needs. Our investigation has also yielded insightful understanding and interpretation of the behavior of the machine learning systems from an end-user software engineering's perspective, rather than a designer's or implementor's perspective, who normally adopts a theoretical approach.Index Terms-Unsupervised machine learning, clustering assessment, clustering validation, metamorphic testing, metamorphic relation, enduser software engineering.1. In mathematics, the convex hull of a set X of points in the Euclidean plane is the smallest convex set that contains X.

show abstract

Section: Metamorphic Testing (Mt)mentioning

confidence: 99%

“…Guidelines were also provided for identifying MRs for both simulation approaches. Case studies [25] showed how MT can help increase users's confidence in the correctness of the simulation models.…”

Section: Metamorphic Testing (Mt)mentioning

confidence: 99%

METTLE: A METamorphic Testing Approach to Assessing and Validating Unsupervised Machine Learning Systems

et al. 2020

View full text Add to dashboard Cite

show abstract

“…MT, now twenty years old, is fast-becoming the first choice for testers of these systems. Since the acceptance for publication of our paper (August 2018) [1], a number of new and exciting directions for MT have come to light, including (but not limited to): metamorphic exploration (an MT-inspired approach that helps users, including testers, better understand and use systems) [2]; MT's application to post-quantum cryptography, as explored at the Loyola University and the American National Institute of Standards and Technology (NIST) [3]; simulation validation using MT [4]; metamorphic robustness testing (revealing hidden defects in the citation database systems Scopus and Web of Science, hence challenging the validity of citation statistics and journal impact factors) [5]; and a reported technique for detecting fatal software faults in the LiDAR obstacle-perception module of self-driving cars, with alarming results reported eight days before Uber's deadly crash in Tempe, Arizona, USA, in March 2018 [6].…”

Section: Recent Mt Developmentsmentioning

confidence: 99%

An Extended Abstract of "Metamorphic Testing: Testing the Untestable"

Segura

Towey

Zhi

et al. 2019

2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC)

View full text Add to dashboard Cite

“…Several studies have been conducted to use MT to test the software for scientific computations and simulations. For instances, in testing simulations, MT has been successfully adopted to test agent-based (ABM) and discrete-event (DES) simulations [26], hybrid ABM and DES systems [11], health care simulation [25], webenabled simulation [1], and simulator platform for self-driving cars [36,41]. In testing scientific software, MT is an effective technique to detect faults in simulation programs for designing nuclear power plants [14], bioinformatics programs [4], epidemiological models [31,33], chemical reaction networks for prototyping nano-scale molecular devices [13], matrix calculation programs [32], solvers for partial differential equations [3], multiple linear regression software [22], ocean modelling [16], storm water management model systems [19], machine learning-based hydro-logical models [40], Monte-Carlo computational programs [10,30], serverless scientific applications [20], as well as other types of scientific software [9,18,19,29].…”

Section: Introductionmentioning

confidence: 99%

Testing Ocean Software with Metamorphic Testing

Luu,

Liu,

Chen

et al. 2022

Preprint

View full text Add to dashboard Cite

Advancing ocean science has a significant impact to the development of the world, from operating a safe navigation for vessels to maintaining a healthy and diverse ocean ecosystem. Various ocean software systems have been extensively adopted for different purposes, for instance, predicting hourly sea level elevation across shorelines, simulating large-scale ocean circulations, as well as integrating into Earth system models for weather forecasts and climate projections. Regardless of their significance, guaranteeing the trustworthiness of ocean software and modelling systems is a long-standing challenge. The testing of ocean software suffers a lot from the so-called oracle problem, which refers to the absence of test oracles mainly due to the nonlinear interactions of multiple physical variables and the high complexity in computation. In the ocean, observed tidal signals are distorted by non-deterministic physical variables, hindering us from knowing the "true" astronomical tidal constituents existing in the timeseries. In this paper, we present how to test tidal analysis and prediction (TAP) software based on metamorphic testing (MT), a simple yet effective testing approach to the oracle problem. In particular, we construct metamorphic relations from the periodic property of astronomical tide, and then use them to successfully detect a real-life defect in an open-source TAP software. We also conduct a series of experiments to further demonstrate the applicability and effectiveness of MT in the testing of TAP software. Our study not only justifies the potential of MT in testing more complex ocean software and modelling systems, but also can be expanded to assess and improve the quality of a broader range of scientific simulation software systems. CCS CONCEPTS• Applied computing → Earth and atmospheric sciences; • Software and its engineering → Software testing and debugging.

show abstract

Increasing Validity of Simulation Models Through Metamorphic Testing

Cited by 19 publications

References 37 publications

METTLE: A METamorphic Testing Approach to Assessing and Validating Unsupervised Machine Learning Systems

METTLE: A METamorphic Testing Approach to Assessing and Validating Unsupervised Machine Learning Systems

An Extended Abstract of "Metamorphic Testing: Testing the Untestable"

Testing Ocean Software with Metamorphic Testing

Contact Info

Product

Resources

About