Using Evolutionary Mutation Testing to improve the quality of test suites

Delgado‐Pérez, Pedro; Medina‐Bulo, Inmaculada; Núñez, Manuel

doi:10.1109/cec.2017.7969365

Cited by 6 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experiments conducted in this paper go a step beyond than previous experiments -where the ability of the technique to find strong mutants was evaluated-by assessing in depth a new methodology that allows us to estimate the extent to which EMT could help improve the test suite. This work connects and extends the results of two previous papers [6,7], providing a comprehensive picture regarding the behavior of EMT, descriptive examples on how to implement the proposed methodology, statistical significance of the results and a discussion comparing the old and the new methodology and the results of EMT, random selection and selective mutation. The paper also includes a list of lessons learned that can be useful for researchers in this field in a foreseeable future.…”

Section: Introductionsupporting

confidence: 69%

See 1 more Smart Citation

Search-based mutant selection for efficient test suite improvement: Evaluation and results

Delgado‐Pérez

Medina‐Bulo

2018

Information and Software Technology

Self Cite

View full text Add to dashboard Cite

Context: Search-based techniques have been applied to almost all areas in software engineering, especially to software testing, seeking to solve hard optimization problems. However, the problem of selecting mutants to improve the test suite at a lower cost has not been explored to the same extent as other problems, such as mutant selection for test suite evaluation or test data generation. Objective: In this paper, we apply search-based mutant selection to enhance the quality of test suites efficiently. Namely, we use the technique known as Evolutionary Mutation Testing (EMT), which allows reducing the number of mutants while preserving the power to refine the test suite. Despite reported benefits of its application, the existing empirical results were derived from a limited number of case studies, a particular set of mutation operators and a vague measure, which currently makes it difficult to determine the real performance of this technique. Method: This paper addresses the shortcomings of previous studies, providing a new methodology to evaluate EMT on the basis of the actual improvement of the test suite achieved by using the evolutionary strategy. We make use of that methodology in new experiments with a carefully selected set of real-world C++ case studies. Results: EMT shows a good performance for most case studies and levels of demand of test suite improvement (around 45% less mutants than random selection in the best case). The results reveal that even a reduced subset of mutants selected with EMT can serve to increase confidence in the test suite, especially in programs with a large set of mutants. Conclusions: These results support the use of search-based techniques to solve the problem of mutant selection for a more efficient test suite refinement. Additionally, we identify some aspects that could foreseeably help enhance EMT.

show abstract

Section: Introductionsupporting

confidence: 69%

“…A graphical explanation of these two operators can be found in [7]. The idea behind these operators is to find similar individuals to those that were selected for reproduction.…”

Section: Selection and Reproductive Operatorsmentioning

confidence: 99%

Search-based mutant selection for efficient test suite improvement: Evaluation and results

Delgado‐Pérez

Medina‐Bulo

2018

Information and Software Technology

Self Cite

View full text Add to dashboard Cite

show abstract

“…The objective is to find a small set of mutation operators that generates a subset of all possible mutants without a major loss of test efficiency. Some of the most recent works in this line of research are previous studies [27][28][29][30]. 4 Parallel Execution [31][32] distributes the mutants among different physical machines, executing test cases in parallel.…”

Section: Cost Reduction Techniquesmentioning

confidence: 99%

Analysing the combination of cost reduction techniques in Android mutation testing

Usaola

Rodríguez-Trujillo

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

When applied to mobile software, mutation testing is particularly costly due to the deployment of the app under test onto the device: if one deployment is made for each generated mutant, the execution time becomes unapproachable. This paper analyses how the combination of different cost reduction techniques improves the execution time of mutation testing in mobile apps. The techniques reviewed and combined are mutant schema, parallel execution and two different ways of executing tests against the mutants (all against all and all against mutants remaining alive), as well as greedy algorithm for reducing the test suite size. This paper also presents a mathematical model of cost reduction and checks its validity with several experiments. Furthermore, the exhaustive and long experimentation has led the authors to compile a set of good practices which are also presented in a set of lessons learned.

show abstract

“…The technique Evolutionary Mutation Testing (EMT) [2] makes use of an EA to generate a reduced subset of interesting mutants for the improvement of functional test suites. In broad terms, EMT searches for mutants that are not killed by the current test suite, which may induce the design of new test cases once they are reviewed.…”

Section: Algorithmmentioning

confidence: 99%

Search-based mutation testing to improve performance tests

Sánchez

Delgado‐Pérez

Medina‐Bulo

et al. 2018

Proceedings of the Genetic and Evolutionary Computation Conference Companion

View full text Add to dashboard Cite

Performance bugs are common and can cause a significant deterioration in the behaviour of a program, leading to costly issues. To detect them and reduce their impact, performance tests are typically applied. However, there is a lack of mechanisms to evaluate the quality of performance tests, causing many of these bugs remain unrevealed. Mutation testing, a fault-based technique to assess and improve test suites, has been successfully studied with func-tional tests. In this paper, we propose the use of mutation testing together with a search-based strategy (evolutionary algorithm) to find mutants that simulate performance issues. This novel approach contributes to enhance the confidence on performance tests while reducing the cost of mutation testing.

show abstract

Using Evolutionary Mutation Testing to improve the quality of test suites

Cited by 6 publications

References 23 publications

Search-based mutant selection for efficient test suite improvement: Evaluation and results

Search-based mutant selection for efficient test suite improvement: Evaluation and results

Analysing the combination of cost reduction techniques in Android mutation testing

Search-based mutation testing to improve performance tests

Contact Info

Product

Resources

About