Testing multiple linear regression systems with metamorphic testing

Luu, Quang-Hung; Lau, Man Fai; Ng, Sebastian; Chen, Tsong Yueh

doi:10.1016/j.jss.2021.111062

Cited by 24 publications

(14 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hereby, we reuse some MRs that have been developed in our previous study for the multiple linear regression (MLR) [22]. Given the source test case = {( 1 , 1 ), ( 2 , 2 ), .…”

Section: Regression-based Mrsmentioning

confidence: 99%

“…The advantage of mutation analysis is to generate a wide range of faults to challenge the effectiveness of the newly developed testing method. In this study, we adopted the mutant-generating tool developed by Luu et al [22], and modified it to suit our purpose of testing Python-based programs.…”

Section: System Under Test and Mutation Analysismentioning

confidence: 99%

“…It contains 206 LOC (in "_solve.py" from line 232 to line 438). Among mutation operators defined in [22] for C/C++ programs, we used five sets to generate mutants for the Python program, as presented in Table 1. We then applied a filtering process to eliminate the equivalent mutants (which refers to the mutants that have the same execution behaviors as the original program): If a mutant always delivers the same output as the UTide on a set of randomly generated test cases, that mutant would be removed.…”

Section: System Under Test and Mutation Analysismentioning

confidence: 99%

“…For instances, in testing simulations, MT has been successfully adopted to test agent-based (ABM) and discrete-event (DES) simulations [26], hybrid ABM and DES systems [11], health care simulation [25], webenabled simulation [1], and simulator platform for self-driving cars [36,41]. In testing scientific software, MT is an effective technique to detect faults in simulation programs for designing nuclear power plants [14], bioinformatics programs [4], epidemiological models [31,33], chemical reaction networks for prototyping nano-scale molecular devices [13], matrix calculation programs [32], solvers for partial differential equations [3], multiple linear regression software [22], ocean modelling [16], storm water management model systems [19], machine learning-based hydro-logical models [40], Monte-Carlo computational programs [10,30], serverless scientific applications [20], as well as other types of scientific software [9,18,19,29]. For example, He et al [14] has found 33 bugs in simulation programs that are used to design and analyze nuclear power plants in a study that adopts MT.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Testing Ocean Software with Metamorphic Testing

Luu,

Liu,

Chen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Advancing ocean science has a significant impact to the development of the world, from operating a safe navigation for vessels to maintaining a healthy and diverse ocean ecosystem. Various ocean software systems have been extensively adopted for different purposes, for instance, predicting hourly sea level elevation across shorelines, simulating large-scale ocean circulations, as well as integrating into Earth system models for weather forecasts and climate projections. Regardless of their significance, guaranteeing the trustworthiness of ocean software and modelling systems is a long-standing challenge. The testing of ocean software suffers a lot from the so-called oracle problem, which refers to the absence of test oracles mainly due to the nonlinear interactions of multiple physical variables and the high complexity in computation. In the ocean, observed tidal signals are distorted by non-deterministic physical variables, hindering us from knowing the "true" astronomical tidal constituents existing in the timeseries. In this paper, we present how to test tidal analysis and prediction (TAP) software based on metamorphic testing (MT), a simple yet effective testing approach to the oracle problem. In particular, we construct metamorphic relations from the periodic property of astronomical tide, and then use them to successfully detect a real-life defect in an open-source TAP software. We also conduct a series of experiments to further demonstrate the applicability and effectiveness of MT in the testing of TAP software. Our study not only justifies the potential of MT in testing more complex ocean software and modelling systems, but also can be expanded to assess and improve the quality of a broader range of scientific simulation software systems. CCS CONCEPTS• Applied computing → Earth and atmospheric sciences; • Software and its engineering → Software testing and debugging.

show abstract

“…Hereby, we reuse some MRs that have been developed in our previous study for the multiple linear regression (MLR) [22]. Given the source test case = {( 1 , 1 ), ( 2 , 2 ), .…”

Section: Regression-based Mrsmentioning

confidence: 99%

Section: System Under Test and Mutation Analysismentioning

confidence: 99%

Section: System Under Test and Mutation Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Testing Ocean Software with Metamorphic Testing

Luu,

Liu,

Chen

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…These testing challenges can be alleviated using the Metamorphic Testing (MT) technique. MT has been successfully applied to test a wide range of sophisticated systems [14], [15], [16], [17], [18]. Its applications have been expanded beyond testing to other purposes such as system understanding, fault localization, program repair, and cybersecurity [19], [14], [20], [21].…”

Section: Introductionmentioning

confidence: 99%

A Sequential Metamorphic Testing Framework for Understanding Automated Driving Systems

Luu¹,

Liu²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Automated driving systems (ADS) are expected to be reliable and robust against a wide range of driving scenarios. Their decisions, first and foremost, must be well understood. Understanding a decision made by ADS is a great challenge, because it is not straightforward to tell whether the decision is correct or not, and how to verify it systematically. In this paper, a Sequential MetAmoRphic Testing (SMART) framework is proposed based on metamorphic testing, a mainstream software testing approach. In metamorphic testing, metamorphic groups are constructed by selecting multiple inputs according to the so-called metamorphic relations, which are basically the system's necessary properties; the violation of certain relations by some corresponding metamorphic groups implies the detection of erroneous system behaviors. The proposed framework makes use of sequences of metamorphic groups to understand ADS behaviors, and is applicable without the need of ground-truth datasets. To demonstrate its effectiveness, the framework is applied to test three ADS models that steer an autonomous car in different scenarios with another car either leading in front or approaching in the opposite direction. The conducted experiments reveal a large number of undesirable behaviors in these top-ranked deep learning models in the scenarios. These counter-intuitive behaviors are associated with how the core models of ADS respond to different positions, directions and properties of the other car in its proximity. Further analysis of the results helps identify critical factors affecting ADS decisions and thus demonstrates that the framework can be used to provide a comprehensive understanding of ADS before their deployment.

show abstract