Automatic Detection and Removal of Ineffective Mutants for the Mutation Analysis of Relational Database Schemas

McMinn, Phil; Wright, Chris J.; McCurdy, Colton J.; Kapfhammer, Gregory M.

doi:10.1109/tse.2017.2786286

Cited by 17 publications

(7 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Effectiveness for mutants is primarily defined in terms of redundacy and equivalence. This approach fails to consider the notion that non-reduntant mutants might be unproductive or that equivalent mutants can be productive [39]. From our experience, reporting equivalent mutants has been a vastly easier problem than reporting unproductive nonreduntant and non-equivalent mutants.…”

Section: Related Workmentioning

confidence: 99%

Practical Mutation Testing at Scale: A view from Google

Petrović

Ivanković

Just

2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem-both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 150,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment; even existing solutions to speed up mutation analysis are insufficient to make it computationally feasible at such a scale. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas:(1) mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base;(2) mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. This paper empirically validates the proposed approach by analyzing its effectiveness in a code-review-based setting, used by more than 24,000 developers on more than 1,000 projects. The results show that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable to industrial settings of any size.

show abstract

Section: Related Workmentioning

confidence: 99%

Practical Mutation Testing at Scale: A view from Google

Petrović

Ivanković

Just

2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Effectiveness for mutants is primarily defined in terms of redundacy and equivalence. This approach fails to consider the notion that non-reduntant mutants might be unproductive or that equivalent mutants can be productive [33]. From our experience, reporting equivalent mutants has been a vastly easier problem than reporting unproductive nonreduntant and non-equivalent mutants.…”

Section: Related Workmentioning

confidence: 99%

Practical Mutation Testing at Scale

Petrović,

Ivanković,

Fraser

et al. 2021

Preprint

View full text Add to dashboard Cite

Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem-both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 500,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment; even existing solutions to speed up mutation analysis are insufficient to make it computationally feasible at such a scale. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas: (1) Mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base; (2) Mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) Mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. This paper empirically validates the proposed approach by analyzing its effectiveness in a code-review-based setting, used by more than 24,000 developers on more than 1,000 projects. The results show that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable up to large-scale industrial settings.

show abstract

“…Recent literature reviews have shown that the research in mutation testing has been steadily increasing over the last two decades [16,17,[25][26][27]. Mutation analysis suffers from the mutant explosion problem, particularly in large FSMs [28][29][30][31].…”

Section: Related Workmentioning

confidence: 99%

Mutant selection by using Fourier expansion

Takan¹,

Ayav²

2020

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

Mutation analysis is a widely used technique to evaluate the effectiveness of test cases in both hardware and software testing. The original model is mutated systematically under certain fault assumptions and test cases are checked against the mutants created to see whether the test cases can detect the faults or not. Mutation analysis is usually a computationally intensive task, particularly in finite state machine (FSM) testing due to a possibly huge amount of mutants. Random selection could be a practical reduction method under the assumption that each mutant is identical in terms of the probability of occurrence of its associating fault. The present study proposes a mutant selection method based on Fourier analysis of Boolean functions. Fourier helps to identify the most effective transitions on the output so that the mutants related to those transitions can be selected. Such mutants are considered more important since they are more likely to be killed. To evaluate the method, test cases are generated by the well-known W method, which has the capability of detecting every potential fault. The original and reduced sets of mutants are compared with respect to their importance values. Evaluations show that the mutants selected by the proposed technique are more effective, which reduces the cost of mutation analysis without sacrificing the performance of the mutation analysis.

show abstract

Automatic Detection and Removal of Ineffective Mutants for the Mutation Analysis of Relational Database Schemas

Cited by 17 publications

References 67 publications

Practical Mutation Testing at Scale: A view from Google

Practical Mutation Testing at Scale: A view from Google

Practical Mutation Testing at Scale

Mutant selection by using Fourier expansion

Contact Info

Product

Resources

About