Crash reproduction via test case mutation: let existing test cases help

Xuan, Jifeng; Xie, Xue‐Jun; Monperrus, Martin

doi:10.1145/2786805.2803206

Cited by 50 publications

(78 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Each point represents an instance (i.e., mutant). Note that some mutants are killed due to "Timed Out", "Memory Error", or "Run Error" 20 . For these three kinds of mutants, PIT does not report which specific tests cause the killing status, and thus we remove them from the total mutant set in this study.…”

Section: Correlation Analysis Of Predictabilitymentioning

confidence: 99%

“…For these three kinds of mutants, PIT does not report which specific tests cause the killing status, and thus we remove them from the total mutant set in this study. 20 http://pitest.org/quickstart/basic concepts/ From the two sub-figures, although there is no obvious linear relationship between the killability and predictability of the mutants, a mutant with higher killability tends to own higher predictability. In other words, mutants that are easier to kill tend to be easier to predict as well.…”

Section: Correlation Analysis Of Predictabilitymentioning

confidence: 99%

“…Except for evaluating test power, mutation testing has also been shown to be suitable for simulating real faults in software testing experimentation [9], [10], [11], localizing faults [12], [13], [14], performing model transformations [15], and guiding test generation [16], [17], [18], [19], [20], [21].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

113

110

View full text Add to dashboard Cite

Abstract-Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60% confidence) when predicting the execution results of the majority of mutants.

show abstract

Section: Correlation Analysis Of Predictabilitymentioning

confidence: 99%

Section: Correlation Analysis Of Predictabilitymentioning

confidence: 99%

See 1 more Smart Citation

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

113

110

View full text Add to dashboard Cite

show abstract

“…The technique uses some mutation operators to iteratively mutate some existing failing tests to derive new test cases potentially useful to localize the specific encountered fault. Similarly, to increases the chance of executing the specific path during crash reproduction, Xuan et al [65] propose an approach based on test case mutation. The approach first selects relevant test cases based on the stack trace in the crash, followed by eliminating assertions in the selected test cases, and finally uses a set of predefined mutation operators to produce new test cases that can help to reproduce the crash.…”

Section: Debugging Effectiveness Improvementmentioning

confidence: 99%

A snowballing literature study on test amplification

Danglot

Vera-Pérez

et al. 2019

Journal of Systems and Software

Self Cite

View full text Add to dashboard Cite

The adoption of agile development approaches has put an increased emphasis on developer testing, resulting in software projects with strong test suites. These suites include a large number of test cases, in which developers embed knowledge about meaningful input data and expected properties in the form of oracles. This article surveys various works that aim at exploiting this knowledge in order to enhance these manually written tests with respect to an engineering goal (e.g., improve coverage of changes or increase the accuracy of fault localization). While these works rely on various techniques and address various goals, we believe they form an emerging and coherent field of research, which we call "test amplification". We devised a first set of papers from DBLP, looking for all papers containing "test" and "amplification" in their title. We reviewed the 70 papers in this set and selected the 4 papers that fit our definition of test amplification. We use these 4 papers as the seed for our snowballing study, and systematically followed the citation graph. This study is the first that draws a comprehensive picture of the different engineering goals proposed in the literature for test amplification. In particular, we note that the goal of test amplification goes far beyond maximizing coverage only. We believe that this survey will help researchers and practitioners entering this new field to understand more quickly and more deeply the intuitions, concepts and techniques used for test amplification.

show abstract

“…Crash reproduction is an important step in debugging field crashes. Therefore, various automated approaches to crash reproduction [3,4,19,20,22,24] have been proposed in the literature. Among these, EvoCrash [22] is a search-based approach, which applies a Guided Genetic Algorithm (GGA) to generate a crashreproducing test.…”

Section: Introductionmentioning

confidence: 99%

Single-objective Versus Multi-objectivized Optimization for Evolutionary Crash Reproduction

Soltani

Derakhshanfar

Panichella

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

EvoCrash is a recent search-based approach to generate a test case that reproduces reported crashes. The search is guided by a fitness function that uses a weighted sum scalarization to combine three different heuristics: (i) code coverage, (ii) crash coverage and (iii) stack trace similarity. In this study, we propose and investigate two alternatives to the weighted sum scalarization: (i) the simple sum scalarization and (ii) the multi-objectivization, which decomposes the fitness function into several optimization objectives as an attempt to increase test case diversity. We implemented the three alternative optimizations as an extension of EvoSuite, a popular search-based unit test generator, and applied them on 33 real-world crashes. Our results indicate that for complex crashes the weighted sum reduces the test case generation time, compared to the simple sum, while for simpler crashes the effect is the opposite. Similarly, for complex crashes, multi-objectivization reduces test generation time compared to optimizing with the weighted sum; we also observe one crash that can be replicated only by multi-objectivization. Through our manual analysis, we found out that when optimizing the original weighted function gets trapped in local optima, optimization for decomposed objectives improves the search for crash reproduction. Generally, while multi-objectivization is under-explored, our results are promising and encourage further investigations of the approach.Partially funded by the EU Horizon 2020 ICT-10-2016-RIA "STAMP" project (No.731529) and the Dutch 4TU project "Big Software on the Run".

show abstract

Crash reproduction via test case mutation: let existing test cases help

Abstract: Releasing a software version Time Crashing when using the software Developer Reporting the bug with crash (stack trace) Generating test cases for automatic crash reproduction Verifying the crash and fixing the bug Tester or end user Computer

Cited by 50 publications

References 8 publications

Predictive Mutation Testing

Predictive Mutation Testing

A snowballing literature study on test amplification

Single-objective Versus Multi-objectivized Optimization for Evolutionary Crash Reproduction

Contact Info

Product

Resources

About