Towards Automated Metamorphic Test Identification for Ocean System Models

Hiremath, Dilip J.; Claus, Martin; Hasselbring, Wilhelm; Rath, Willi

doi:10.1109/met52542.2021.00014

Cited by 8 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In ocean science, Hiremath et al [16] conducted the first study to adopt MT to validate ocean systems. They developed a machine learning algorithm that is able to identify MRs automatically by optimizing a predefined cost function.…”

Section: Related Workmentioning

confidence: 99%

“…For instances, in testing simulations, MT has been successfully adopted to test agent-based (ABM) and discrete-event (DES) simulations [26], hybrid ABM and DES systems [11], health care simulation [25], webenabled simulation [1], and simulator platform for self-driving cars [36,41]. In testing scientific software, MT is an effective technique to detect faults in simulation programs for designing nuclear power plants [14], bioinformatics programs [4], epidemiological models [31,33], chemical reaction networks for prototyping nano-scale molecular devices [13], matrix calculation programs [32], solvers for partial differential equations [3], multiple linear regression software [22], ocean modelling [16], storm water management model systems [19], machine learning-based hydro-logical models [40], Monte-Carlo computational programs [10,30], serverless scientific applications [20], as well as other types of scientific software [9,18,19,29]. For example, He et al [14] has found 33 bugs in simulation programs that are used to design and analyze nuclear power plants in a study that adopts MT.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Testing Ocean Software with Metamorphic Testing

Luu,

Liu,

Chen

et al. 2022

Preprint

View full text Add to dashboard Cite

Advancing ocean science has a significant impact to the development of the world, from operating a safe navigation for vessels to maintaining a healthy and diverse ocean ecosystem. Various ocean software systems have been extensively adopted for different purposes, for instance, predicting hourly sea level elevation across shorelines, simulating large-scale ocean circulations, as well as integrating into Earth system models for weather forecasts and climate projections. Regardless of their significance, guaranteeing the trustworthiness of ocean software and modelling systems is a long-standing challenge. The testing of ocean software suffers a lot from the so-called oracle problem, which refers to the absence of test oracles mainly due to the nonlinear interactions of multiple physical variables and the high complexity in computation. In the ocean, observed tidal signals are distorted by non-deterministic physical variables, hindering us from knowing the "true" astronomical tidal constituents existing in the timeseries. In this paper, we present how to test tidal analysis and prediction (TAP) software based on metamorphic testing (MT), a simple yet effective testing approach to the oracle problem. In particular, we construct metamorphic relations from the periodic property of astronomical tide, and then use them to successfully detect a real-life defect in an open-source TAP software. We also conduct a series of experiments to further demonstrate the applicability and effectiveness of MT in the testing of TAP software. Our study not only justifies the potential of MT in testing more complex ocean software and modelling systems, but also can be expanded to assess and improve the quality of a broader range of scientific simulation software systems. CCS CONCEPTS• Applied computing → Earth and atmospheric sciences; • Software and its engineering → Software testing and debugging.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Testing Ocean Software with Metamorphic Testing

Luu,

Liu,

Chen

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…If the violation is false, the test helps retrain the model. [97,98] predict metamorphic relations for ocean modeling. The RL approach poses relations, evaluates whether they hold, and attempts to minimize a cost function based on the validity of the set of proposed relations.…”

Section: Test Oracle Generationmentioning

confidence: 99%

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

“…HiRemath et al leveraged the metamorphic test to the ocean system modelling program, using machine learning to automatically generate metamorphic relationships (Hiremath et al, 2021).…”

Section: Machine Learning Algorithmsmentioning

confidence: 99%

“…Usman et al (2020) proposed the TestMC framework which leverages the metamorphic testing to verify the industrial‐strength model counter to improve the detection efficiency. HiRemath et al leveraged the metamorphic test to the ocean system modelling program, using machine learning to automatically generate metamorphic relationships (Hiremath et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Syntax‐based metamorphic relation prediction via the bagging framework

Yin

Ruan

et al. 2021

Expert Systems

View full text Add to dashboard Cite

Software testing is an indispensable part of the software engineering industry, which guarantees product reliability and safety. Traditional testing approaches face the testing Oracle problem, they are difficult to construct the expected outputs with the increasing of program complexity. As a result, metamorphic testing, which tests the program by examining the relationship between the execution results, is proposed. However, existing manual metamorphic relation construction requires huge effects of domain experts, and automatic methods are unstable and inefficient due to the insufficient software feature mining. Hence, we proposed a multi‐dimensional program structure‐based metamorphic relation prediction approach, which is composed of feature extraction and prediction model building. In the feature extraction stage, the testing program is converted to multiple intermediate structures (such as control flow graphs and abstract syntax trees) to explore its features. In the prediction model building stage, the extracted feature set is used as the training set, and a novel semi‐supervised support vector machine‐bagging‐K‐nearest neighbors algorithm is designed to train the prediction model. Besides, a two‐phase hybrid granularity search algorithm is proposed to improve the prediction performance by selecting the optimal number of weak classifiers. Compared with existing approaches, our proposed model can improve the accuracy by around 14%.

show abstract

Towards Automated Metamorphic Test Identification for Ocean System Models

Cited by 8 publications

References 16 publications

Testing Ocean Software with Metamorphic Testing

Testing Ocean Software with Metamorphic Testing

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Syntax‐based metamorphic relation prediction via the bagging framework

Contact Info

Product

Resources

About