Coverage-based quality metric of mutation operators for test suite improvement

Abstract

The choice of mutation operators is a fundamental aspect in mutation testing to guide the tester to an effective test suite. Designing a set of mutation operators is subject to a trade-off between effectiveness and computational cost: a larger mutation population might uncover more faults, but will take longer to analyse. With the aim of resolving this trade-off, several authors have defined an assortment of metrics to determine the most valuable operators. In this work, we extend an existing quality metric by incorporating an additional source of data, coverage information, and therefore investigate the extent to which mutants that often covered but rarely killed can improve the evaluation of mutation operators for the refinement of the test suite. As a case study, we analyse C++ class-level operators based on the new coveragebased quality metric to assess whether the original metric is enhanced. The results when selecting the best-valued operators show that this metric has great potential to help the tester in finding effective mutation operators. In comparison with the metric from which is derived, the use of coverage data allows to reduce the number of mutants but often losing fewer test cases and, in addition, retaining those that seem hard to design.

Novel Metrics for Mutation Analysis

“…Sánchez et al examined the utility of mutation testing to evaluate and enhance performance testing [14]. Delgado-Pérez et al investigated the viability of performing performance mutation testing at the source code level for general-purpose programming languages [15]. Zhu et al proposed a collection of "mutation score antipatterns" that enable software engineers to refactor existing code or add tests to increase the mutation score [16].…”

“…To identify the most valuable operators, Delgado-Pérez et al defined metrics [15]. Coverage data reduces the number of mutants at the expense of poor performance.…”