On the limits of mutation reduction strategies

Gopinath, Rahul; Alipour, Mohammad Amin; Ahmed, Iftekhar; Jensen, Carlos; Groce, Alex

doi:10.1145/2884781.2884787

Cited by 38 publications

(33 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While questions of this nature could be executed through, for instance, either a manual analysis or a bespoke program, this paper presents mrstudyr, a tool that makes it easy to effectively pose and answer questions about methods for mutant reduction. Currently, mrstudyr can perform a widely-studied mutant reduction technique: mutant sampling [4], [5], [8]. Due to its extensible design, the mrstudyr tool could be extended to perform specific reduction techniques such as E-selective [24], where mutants are only created by the following operators: ABS, UOI, LCR, AOR, and ROR [8].…”

Section: A Objectivesmentioning

confidence: 99%

“…Mutation testing is a widely-recognized technique for assessing the quality of a test suite [4]. While there are many potential faults for a program, mutation testing focuses on those that are "close" to the correct version, with the expectation that they will be representative of all faults [5].…”

Section: Introductionmentioning

confidence: 99%

“…Although mutation testing effectively requires tests to detect faults, it has significant drawbacks in its computational cost and the amount of necessary human interaction, often making it impractical to use [4], [7], [8]. A major computational cost of mutation testing comes from executing each test case in a test suite for the many generated mutants [9], [10].…”

Section: Introductionmentioning

confidence: 99%

“…In addition to being conceptually simple [4], mutant sampling has been experimentally shown to outperform other more sophisticated methods [13]. Two sub-techniques within mutant sampling are called uniform random sampling and sampling over operators [4], [8], [14].…”

Section: Introductionmentioning

confidence: 99%

“…Two sub-techniques within mutant sampling are called uniform random sampling and sampling over operators [4], [8], [14]. For both of these sub-techniques, a threshold for the maximum percentage of selected mutants is set to x, which is then either applied to the entire set of mutants or to each set of mutants produced by an operator [4], [8], [14].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques

McCurdy

McMinn

Kapfhammer

2016

2016 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

Abstract-Mutation testing is a well-known method for measuring a test suite's quality. However, due to its computational expense and intrinsic difficulties (e.g., detecting equivalent mutants and potentially checking a mutant's status for each test), mutation testing is often challenging to practically use. To control the computational cost of mutation testing, many reduction strategies have been proposed (e.g., uniform random sampling over mutants). Yet, a stand-alone tool to compare the efficiency and effectiveness of these methods is heretofore unavailable. Since existing mutation testing tools are often complex and languagedependent, this paper presents a tool, called mrstudyr, that enables the "retrospective" study of mutant reduction methods using the data collected from a prior analysis of all mutants. Focusing on the mutation operators and the mutants that they produce, the presented tool allows developers to prototype and evaluate mutant reducers without being burdened by the implementation details of mutation testing tools. Along with describing mrstudyr's design and overviewing the experimental results from using it, this paper inaugurates the public release of this open-source tool.

show abstract

Section: A Objectivesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques

McCurdy

McMinn

Kapfhammer

2016

2016 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

show abstract

Assessment of C++ object‐oriented mutation operators: A selective mutation approach

Delgado‐Pérez

Segura

Medina‐Bulo

2017

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Mutation testing is an effective but costly testing technique. Several studies have observed that some mutants can be redundant and therefore removed without affecting its effectiveness. Similarly, some mutants may be more effective than others in guiding the tester on the creation of high‐quality test cases. On the basis of these findings, we present an assessment of C++ class mutation operators by classifying them into 2 rankings: the first ranking sorts the operators on the basis of their degree of redundancy and the second regarding the quality of the tests they help to design. Both rankings are used in a selective mutation study analysing the trade‐off between the reduction achieved and the effectiveness when using a subset of mutants. Experimental results consistently show that leveraging the operators at the top of the 2 rankings, which are different, lead to a significant reduction in the number of mutants with a minimum loss of effectiveness.

show abstract

Mutation operators for feature‐oriented software product lines

Krüger

Al-Hajjaji

Leich

et al. 2018

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Mutation testing is an approach to assess the quality of test cases. Mutants are modified versions of a system that ideally compose faulty behaviour. Test cases for a system are effective if they kill these mutants. For software product lines, several works have addressed mutation testing to inject variability faults, which may only exist in some variants. These works focus on variability models or specific implementation techniques. In contrast, feature‐oriented programming has been rarely investigated, wherefore, we (1) derive corresponding mutation operators, (2) investigate the feasibility of our proposed and conventional operators on 4 software product lines, and (3) discuss open challenges in mutation testing of software product lines. The results show that our proposed operators are suitable to cause variability faults and extend the capabilities of conventional operators. Nonetheless, mutation testing of software product lines is comparably expensive, due to a high number of variants and mutants—resulting in equivalence and redundancy.

show abstract

On the limits of mutation reduction strategies

Cited by 38 publications

References 36 publications

mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques

mrstudyr: Retrospectively Studying the Effectiveness of Mutant Reduction Techniques

Assessment of C++ object‐oriented mutation operators: A selective mutation approach

Mutation operators for feature‐oriented software product lines

Contact Info

Product

Resources

About