Self-Checked Metamorphic Testing of an Image Processing Program

Abstract

Metamorphic testing is an effective technique for testing systems that do not have test oracles, for which it is practically impossible to know the correct output of an arbitrary test input. In metamorphic testing, instead of checking the correctness of a test output, the satisfaction of metamorphic relation among test outputs is checked. If a violation of the metamorphic relation is found, the system implementation must have some defects. However, a randomly or accidently generated incorrect output may satisfy a metamorphic relation as well. Therefore, checking only metamorphic relations is not good enough to ensure the testing quality. In this paper, we propose a self-checked metamorphic testing approach, which integrates structural testing into metamorphic testing, to detect subtle defects in a system implementation. In our approach, metamorphic testing results are further verified by test coverage information, which is automatically produced during the metamorphic testing. The effectiveness of the approach has been investigated through testing an image processing program.

A Survey on Metamorphic Testing

“…They defined the "difference among executions" as any aspects of program runs (e.g., paths traversed). This observation has been confirmed by several later studies [9], [41], [42], [43], [44], [45]. In particular, Asrafi et al [46] hypothesised that the higher the combined code coverage of the source and follow-up test cases, the more different are the executions, and the more effective is the metamorphic relation.…”

“…Ding et al [43] proposed a method named Self-Checked Metamorphic Testing (SCMT) combining metamorphic testing and structural testing. SCMT checks the code coverage of source and follow-up test cases during test execution to evaluate the quality of metamorphic relations.…”

“…Ding et al [105] presented a case study on the detection of faults in a Monte Carlo modelling program for the simulation of photon propagation. Based on their previous work [43], the authors used code coverage criteria to guide the selection of effective metamorphic relations and the creation of test cases. Murphy et al [76] proposed using metamorphic relations to systematically test health care simulation programs, and presented a case study with two real-world simulators and mutation testing.…”

A Machine Learning Approach for Developing Test Oracles for Testing Scientific Software

“…Our preliminary result has shown the cells can be classified with the accuracy as high as the measured diffraction images. However, we haven't completed the experiment for all 4 data that the major differences between PC3 and PCS cells are in the cell volume. The PC3 cells are almost two times bigger than PCS in the cell volume, while the PCS cells have a larger volume ratio of mitochondria-to-cell than PC3.…”

“…However, the quality of metamorphic testing is highly depended on MRs. Due to the grand challenge to develop strong MRs, many MRs used for testing a complex software system are relatively too weak to ensure the testing quality. It is important to define a set of criteria for evaluating the adequacy of MRs. Then the test execution and evaluation results shall be used for guiding the generation of adequate MRs [4]. The approach introduced in this paper includes a framework for the development of MRs and tests, and a strategy for refining the relations and tests though mining the test execution and evaluation results with machine learning algorithms.…”