On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions

Cao, Yuxiang; Zhi, Zhou; Chen, Tsong Yueh

doi:10.1109/qsic.2013.43

Cited by 33 publications

(26 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their study on two subject programs showed a strong correlation between coverage and fault-detection effectiveness in one of the two. In a similar study, Cao et al [47] assessed the relation between fault-detection effectiveness of metamorphic relations and test case dissimilarity. An extensive experiment with 83 faulty programs and 7 distance metrics between the execution profiles of source and follow-up test cases revealed a strong and statistically significant correlation between the fault-detection capability of metamorphic relations and the distance among test cases, in particular when using branch coverage Manhattan distance [48].…”

Section: Properties Of Good Metamorphic Relationsmentioning

confidence: 99%

A Survey on Metamorphic Testing

Segura

Fraser

Sánchez

et al. 2016

IIEEE Trans. Software Eng.

405

230

View full text Add to dashboard Cite

Abstract-A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output "morphs" into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.

show abstract

Section: Properties Of Good Metamorphic Relationsmentioning

confidence: 99%

A Survey on Metamorphic Testing

Segura

Fraser

Sánchez

et al. 2016

IIEEE Trans. Software Eng.

405

230

View full text Add to dashboard Cite

show abstract

“…Suppose that MR 1 's MTG covers the first and second path and thereby executes statements s 1 and s 2 , and that MR 2 's MTG covers the first and third path and so covers all three statements. This demonstrates that an MR's MTG can obtain greater coverage, if the paths that are traversed by each of its test cases are different (Cao et al 2013). Several guidelines have been proposed to design MRs to have such MTGs.…”

Section: Coveragementioning

confidence: 93%

“…Several guidelines have been proposed to design MRs to have such MTGs. For example, white-box analysis techniques (Dong et al 2013), or coverage information generated by regression testing (Cao et al 2013) could assist in the identification of MRs that have MTGs with different test cases. MRs that use a similar strategy to the SUT tend to have MTGs that have similar source and follow up test cases (Mayer and Guderlei 2006), and thus should be avoided.…”

Section: Coveragementioning

confidence: 99%

“…MRs that use a similar strategy to the SUT tend to have MTGs that have similar source and follow up test cases (Mayer and Guderlei 2006), and thus should be avoided. Different MRs can have different MTG sizes (Cao et al 2013). It seems intuitive that MRs that have MTGs that consist of a larger number of test cases are more likely to have test cases that traverse dissimilar paths.…”

Section: Coveragementioning

confidence: 99%

“…However, MRs are necessary, but not sufficient (Chen et al 2003b); they are not effective for all fault types, e.g. coincidentally correct faults (Cao et al 2013;Yoo 2010).…”

Section: Faultsmentioning

confidence: 99%

See 2 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

The terms "Oracle Problem" and "Non-testable system" interchangeably refer to programs in which the application of test oracles is infeasible. Test oracles are an integral part of conventional testing techniques; thus, such techniques are inoperable in these programs. The prevalence of the oracle problem has inspired the research community to develop several automated testing techniques that can detect functional software faults in such programs. These techniques include N-Version testing, Metamorphic Testing, Assertions, Machine Learning Oracles, and Statistical Hypothesis Testing. This paper presents a Mapping Study that covers these techniques. The Mapping Study presents a series of discussions about each technique, from different perspectives, e.g. effectiveness, efficiency, and usability. It also presents a comparative analysis of these techniques in terms of these perspectives. Finally, potential research opportunities within the non-testable systems problem domain are highlighted within the Mapping Study. We believe that the aforementioned discussions and comparative analysis will be invaluable for new researchers that are attempting to familiarise themselves with the field, and be a useful resource for practitioners that are in the process of selecting an appropriate technique for their context, or deciding how to apply their selected technique. We also believe that our own insights, which are embedded throughout these discussions and the comparative analysis, will be useful for researchers that are already accustomed to the field. It is our hope that the potential research opportunities that have been highlighted by the Mapping Study will steer the direction of future research endeavours.

show abstract

Metamorphic relation prioritization for effective regression testing

Srinivasan

Kanewala

2022

Software Testing Verif & Rel

View full text Add to dashboard Cite

Metamorphic testing (MT) is widely used for testing programs that face the oracle problem. It uses a set of metamorphic relations (MRs), which are relations among multiple inputs and their corresponding outputs to determine whether the program under test is faulty. Typically, MRs vary in their ability to detect faults in the program under test, and some MRs tend to detect the same set of faults. In this paper, we propose approaches to prioritize MRs to improve the efficiency and effectiveness of MT for regression testing. We present two MR prioritization approaches: (i) fault-based and (ii) coverage-based. To evaluate these MR prioritization approaches, we conduct experiments on three complex open-source software systems. Our results show that the MR prioritization approaches developed by us significantly outperform the current practice of executing the source and follow-up test cases of the MRs in an adhoc manner in terms of fault detection effectiveness. Further, fault-based MR prioritization leads to reducing the number of source and follow-up test cases that needs to be executed as well as reducing the average time taken to detect a fault, which would result in saving time and cost during the testing process.

show abstract

On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions

Cited by 33 publications

References 20 publications

A Survey on Metamorphic Testing

A Survey on Metamorphic Testing

A mapping study on testing non-testable systems

Metamorphic relation prioritization for effective regression testing

Contact Info

Product

Resources

About