An empirical study on the application of mutation testing for a safety-critical industrial software system

Ramler, Rudolf; Wetzlmaier, Thomas; Klammer, Claus

doi:10.1145/3019612.3019830

Cited by 24 publications

(19 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This assumption comes with the use of mutation testing. We believe that it holds given the empirical evidence provided by recent studies (Titcheu Chekam et al 2017;Petrovic and Ivankovic 2018;Ramler et al 2017;Papadakis et al 2018b;Just et al 2014b). Finally, we assume that fault revelation utility can be captured by static features such as the ones used in this study.…”

Section: Working Assumptionssupporting

confidence: 51%

Selecting fault revealing mutants

et al. 2019

View full text Add to dashboard Cite

Mutant selection refers to the problem of choosing, among a large number of mutants, the (few) ones that should be used by the testers. In view of this, we investigate the problem of selecting the fault revealing mutants, i.e., the mutants that are killable and lead to test cases that uncover unknown program faults. We formulate two variants of this problem: the fault revealing mutant selection and the fault revealing mutant prioritization. We argue and show that these problems can be tackled through a set of 'static' program features and propose a machine learning approach, named FaRM, that learns to select and rank killable and fault revealing mutants. Experimental results involving 1,692 real faults show the practical benefits of our approach in both examined problems. Our results show that FaRM achieves a good trade-off between application cost and effectiveness (measured in terms of faults revealed). We also show that FaRM outperforms all the existing mutant selection methods, i.e., the random mutant sampling, the selective mutation and defect prediction (mutating the code areas pointed by defect prediction). In particular, our results show that with respect to mutant selection, our approach reveals 23% to 34% more faults than any of the baseline methods, while, with respect to mutant prioritization, it achieves higher average percentage of revealed faults with a median difference between 4% and 9% (from the random mutant orderings).

show abstract

Section: Working Assumptionssupporting

confidence: 51%

Selecting fault revealing mutants

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Similarly, the studies of Baker and Habli [57], Ramler et al [64] and [74] have shown that mutation testing provides valuable guidance towards improving existing test suites.…”

Section: The Relations Between Mutants and Fault Revelationmentioning

confidence: 95%

Mutation Testing Advances: An Analysis and Survey

Papadakis

Kintis

Zhang

et al. 2019

Advances in Computers

321

326

View full text Add to dashboard Cite

Mutation testing realises the idea of using artificial defects to support testing activities. Mutation is typically used as a way to evaluate the adequacy of test suites, to guide the generation of test cases and to support experimentation.Mutation has reached a maturity phase and gradually gains popularity both in academia and in industry. This chapter presents a survey of recent advances, over the past decade, related to the fundamental problems of mutation testing and sets out the challenges and open problems for the future development of the method. It also collects advices on best practices related to the use of mutation in empirical studies of software testing. Thus, giving the reader a 'mini-handbook'-style roadmap for the application of mutation testing as experimental methodology.

show abstract

“…Furthermore, all these studies have been assessed by employing two main evaluation metrics. These are either some form of correlation analysis between mutant kills and fault detection ratios (studies with references [3,4,10,24,28]) or the fault detection rate at the higher mutation score levels (studies [8,14,37,38,40]). Therefore, it is unclear what the relationship between these two evaluation metrics is, and what the relationship implies for the actual application of mutation testing.…”

Section: Mutants and Real Faultsmentioning

confidence: 99%

Are mutation scores correlated with real fault detection?

Papadakis

Shin

Yoo

et al. 2018

Proceedings of the 40th International Conference on Software Engineering

View full text Add to dashboard Cite

Empirical validation of software testing studies is increasingly relying on mutants. This practice is motivated by the strong correlation between mutant scores and real fault detection that is reported in the literature. In contrast, our study shows that correlations are the results of the confounding effects of the test suite size. In particular, we investigate the relation between two independent variables, mutation score and test suite size, with one dependent variable the detection of (real) faults. We use two data sets, CoreBench and De-fects4J, with large C and Java programs and real faults and provide evidence that all correlations between mutation scores and real fault detection are weak when controlling for test suite size. We also find that both independent variables significantly influence the dependent one, with significantly better fits, but overall with relative low prediction power. By measuring the fault detection capability of the top ranked, according to mutation score, test suites (opposed to randomly selected test suites of the same size), we find that achieving higher mutation scores improves significantly the fault detection. Taken together, our data suggest that mutants provide good guidance for improving the fault detection of test suites, but their correlation with fault detection are weak. CCS CONCEPTS • Software and its engineering → Software testing and debugging;

show abstract

An empirical study on the application of mutation testing for a safety-critical industrial software system

Cited by 24 publications

References 12 publications

Selecting fault revealing mutants

Selecting fault revealing mutants

Mutation Testing Advances: An Analysis and Survey

Are mutation scores correlated with real fault detection?

Contact Info

Product

Resources

About