Mutation Testing Advances: An Analysis and Survey

Papadakis, Mike; Kintis, Marinos; Zhang, Jie M.; Jia, Yunyi; Traon, Yves Le; Harman, Mark

doi:10.1016/bs.adcom.2018.03.015

Cited by 346 publications

(380 citation statements)

References 304 publications

Supporting

Mentioning

343

Contrasting

Unclassified

Order By: Relevance

“…Thus, we collect only the mutation operation type in this study. For the various static-metric-related features (e.g., depInheritance and numChildren), we implement our own tool based on the abstract syntax tree (AST) analysis provided by the Eclipse JDT toolkit 6 (as shown in the red square in Figure 2). To obtain the remaining features related to the oracle information (i.e., numMutantAssertion and numClassAssertion), we develop our own tool based on bytecode analysis using the ASM bytecode manipulation and analysis framework 7 .…”

Section: Feature Collectionmentioning

confidence: 99%

See 1 more Smart Citation

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

Self Cite

113

110

View full text Add to dashboard Cite

Abstract-Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60% confidence) when predicting the execution results of the majority of mutants.

show abstract

Section: Feature Collectionmentioning

confidence: 99%

“…Mutation testing [4], [5], [6] is a methodology for checking test power that addresses the short-comings of coverage. By contrast with code coverage, mutation testing generates faulty programs and checks whether the test suite can detect these faults.…”

Section: Introductionmentioning

confidence: 99%

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

Self Cite

113

110

View full text Add to dashboard Cite

show abstract

“…Their approach, quite different than MUSE, MUSEUM Metallaxis, or our own, uses mutants to mark likely causal relationships in call graphs. A recent survey of advances in mutation testing places all of these approaches using mutants in fault localization within the larger context of mutation testing research .…”

Section: Related Workmentioning

confidence: 99%

Using mutants to help developers distinguish and debug (compiler) faults

Holmes

Groce

2020

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Measuring the distance between two program executions is a fundamental problem in dynamic analysis of software and useful in many test generation and debugging algorithms. This paper proposes a metric for measuring distance between executions and specializes it to an important application: determining similarity of failing test cases for the purpose of automated fault identification and localization in debugging based on automatically generated compiler tests. The metric is based on a causal concept of distance where executions are similar to the degree that changes in the program itself, introduced by mutation, cause similar changes in the correctness of the executions. Specifically, if two failing test cases for a compiler can be made to pass by applying the same mutant, those two tests are more likely to be due to the same fault. We evaluate our metric using more than 50 faults and 2,800 test cases for two widely used real‐world compilers and demonstrate improvements over state‐of‐the‐art methods for fault identification and localization. A simple operator selection approach to reducing the number of mutants can reduce the cost of our approach by 70%, while producing a gain in fault identification accuracy. We additionally show that our approach, although devised for compilers, is applicable as a conservative fault localization algorithm for other types of programs and can help triage certain types of crashes found in fuzzing non‐compiler programs more effectively than a state‐of‐the‐art technique.

show abstract

“…We performed an expert literature review by considering the papers collected in the recent survey of mutation testing [8]. Our analysis revealed the following two classes of indicators:…”

Section: Mutant Quality Indicatorsmentioning

confidence: 99%

Mutant Quality Indicators

Papadakis

Chekam

Traon

2018

2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

Self Cite

View full text Add to dashboard Cite

Abstract-The question of which are the valuable mutants has received little attention in mutation testing literature. Naturally, the choice of mutants impacts the quality of the performed analysis and has the potential of changing the conclusions of empirical studies. To this end, we collect definitions related to mutant quality indicators and analyses their relations. We identify two classes of indicators, related to individual mutants (Fault Revealing, Subsuming, Hard-to-kill and Stubborn) and to mutant sets (disjoint/dominator and distinguished). We analyse a large set of mutants from 3,902 (real) faulty program versions, belonging to 40 fault classes, collected from an on-line programming contest. Our analysis categorises mutants as valuable, according to the studied quality indicators, profiles their types and examines the relations between them. Our results suggest that there is a large disagreement between the indicators and that the connection between mutant type, its quality and its ability to reveal faults is weak. Additionally, our paper reveal that the ability of mutants to uncover faults differs significantly across the different fault classes and that some mutant types are well linked (or completely disconnected) to specific fault classes.

show abstract

Mutation Testing Advances: An Analysis and Survey

Cited by 346 publications

References 304 publications

Predictive Mutation Testing

Predictive Mutation Testing

Using mutants to help developers distinguish and debug (compiler) faults

Mutant Quality Indicators

Contact Info

Product

Resources

About