Feedback-Directed Metamorphic Testing

Abstract

Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a set of metamorphic relations among multiple software inputs and their expected outputs, metamorphic testing not only provides a test case generation strategy by constructing new (or follow-up) test cases from some original (or source) test cases, but also a test result verification mechanism through checking the relationship between the outputs of source and follow-up test cases. Many efforts have been made to further improve the cost-effectiveness of metamorphic testing from different perspectives. Some studies attempted to identify “good” metamorphic relations, while other studies were focused on applying effective test case generation strategies especially for source test cases. In this paper, we propose improving the cost-effectiveness of metamorphic testing by leveraging the feedback information obtained in the test execution process. Consequently, we develop a new approach, namely feedback-directed metamorphic testing, which makes use of test execution information to dynamically adjust the selection of metamorphic relations and selection of source test cases. We conduct an empirical study to evaluate the proposed approach based on four laboratory programs, one GNU program, and one industry program. The empirical results show that feedback-directed metamorphic testing can use fewer test cases and take less time than the traditional metamorphic testing for detecting the same number of faults. It is clearly demonstrated that the use of feedback information about test execution does help enhance the cost-effectiveness of metamorphic testing. Our work provides a new perspective to improve the efficacy and applicability of metamorphic testing as well as many other software testing techniques.

Identifying metamorphic relations: A data mutation directed approach

“…Previous works also show that a set of diverse MRs can result in a higher fault detection effectiveness. 56,57 A plausible reason is that the more diverse MRs tend to cover more independent properties of a SUT and thus have a bigger chance of detecting faults that violates these properties in one way or another. Considering this advantage, MR composition will be leveraged in our proposed MR identification technique.…”

“…Clearly, through the MR composition, one can derive a lot of composite MRs even with a small number of identified component MRs, which significantly reduces the efforts when a large number of diverse MRs are expected. Previous works also show that a set of diverse MRs can result in a higher fault detection effectiveness 56,57 . A plausible reason is that the more diverse MRs tend to cover more independent properties of a SUT and thus have a bigger chance of detecting faults that violates these properties in one way or another.…”