The use of mutation in testing experiments and its sensitivity to external threats

Abstract

Mutation analysts has emerged as a standard approach for empirical assessment of testing techniques. The test practitioners decide about cost-effectiveness of testing strategies based on the number of mutants the testing techniques detect. Though fundamental rigor to empirical software testing, the use of mutants in the absence of real-world faults has raised the concern of whether mutants and real faults exhibit similar properties. This paper revisits this important concern and disseminates interesting findings regarding mutants and whether these synthetic faults can predict fault detection ability of test suites. The results of controlled experiments conducted in this paper show that mutation when used in testing experiments is highly sensitive to external threats caused by some influential factors including mutation operators, test suite size, and programming languages. This paper raises the awareness message of the use of mutation in testing experiment and suggests that any interpretation or generalization of experimental findings based on mutation should be justified according to the influential factors involved.

Are mutants a valid substitute for real faults in software testing?

“…Request permissions from Permissions@acm.org. real faults [1,8,27]. Our work differs from these previous studies in four main aspects.…”

“…Namin and Kakarla [27] later replicated the work of Andrews et al [1], used a different mutation testing tool (Proteum), and came to a different conclusion: they found that the correlation between the mutation score and the real fault detection rate for Space was weak. They also extended the work to five Java classes from the standard library, ranging from 197 to 895 lines of code.…”

PIT: a practical mutation testing tool for Java (demo)

“…However, in practice mutation is sensitive to the underlying mutants that it is using. In other words, the set of the realised operators can have a major impact on both scalability [9] and effectiveness [8] of the technique. Therefore, it is mandatory to equip mutation tools with a comprehensive set of mutants that can adequately measure test thoroughness and at the same time is practical.…”

CoREBench: studying complexity of regression errors

“…Namin et al [21] caution that this insight is highly sensitive to external threats mentioning several influential factors that must be accounted for. In the present paper we have investigated not the detectability but the complexity of regression errors and found that the complexity of regression errors resulting from seeded faults is different from that of actual regression errors.…”

“…Since the method traverseChange is a recursive traversal algorithm, we mark the visited vertices as such (lines [14][15]. If any basic block bb transitively following c is changed, then add an edge from c to bb (lines [16][17][18][19][20][21]. If c or any transitively following basic block, transitively calls a changed method m ′ , continue traversal from the f irst basic block in the CF G ′ of m ′ (lines [23][24][25][26].…”