Operator-based and random mutant selection: Better together

Zhang, Lingming; Gligoric, Milos; Marinov, Darko; Khurshid, Sarfraz

doi:10.1109/ase.2013.6693070

Cited by 74 publications

(88 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, they chose only concurrent programs as they explored selective mutation for concurrent mutation operators. Zhang et al [40] used 11 real-world projects whose number of lines of code is from 2681 to 36910, but they did not systematically study the scalability problem in selective mutation testing.…”

Section: Motivationmentioning

confidence: 99%

“…Following the same procedures of previous studies on mutation testing [32], [42], [40], in this empirical study, the mutants that cannot be killed by any test case from the given test suite are taken as equivalent mutants 9 . By removing these equivalent mutants, we got the set of non-equivalent mutants, which can be killed by the given test suite.…”

Section: B Subjectsmentioning

confidence: 99%

See 1 more Smart Citation

An Empirical Study on the Scalability of Selective Mutation Testing

Zhang

Zhu

Hao

et al. 2014

2014 IEEE 25th International Symposium on Software Reliability Engineering

View full text Add to dashboard Cite

Software testing plays an important role in ensuring software quality by running a program with test suites. Mutation testing is designed to evaluate whether a test suite is adequate in detecting faults. Due to the expensive cost of mutation testing, selective mutation testing was proposed to select a subset of mutants whose effectiveness is similar to the whole set of generated mutants. Although selective mutation testing has been widely investigated in recent years, many people still doubt whether it can suit well for large programs. To study the scalability of selective mutation testing, we systematically explore how the program size impacts selective mutation testing through four projects (including 12 versions all together). Based on the empirical study, for programs smaller than 16 KLOC, selective mutation testing has surprisingly good scalability. In particular, for a program whose number of lines of executable code is E, the number of mutants used in selective mutation testing is proportional to E c , where c is a constant whose value is between 0.05 and 0.25.

show abstract

Section: Motivationmentioning

confidence: 99%

Section: B Subjectsmentioning

confidence: 99%

An Empirical Study on the Scalability of Selective Mutation Testing

Zhang

Zhu

Hao

et al. 2014

2014 IEEE 25th International Symposium on Software Reliability Engineering

View full text Add to dashboard Cite

show abstract

“…This is a common practice in this kind of experiment [1], [4], [5], [55], because of the inherent underlying decidability problem. However, it is also a potential limitation of our study, like others.…”

Section: Threats To Validitymentioning

confidence: 99%

An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption

Chekam

Papadakis

Traon

et al. 2017

2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

107

109

View full text Add to dashboard Cite

Abstract-Many studies suggest using coverage concepts, such as branch coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar coverage for both faulty and fixed ('clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of coverage are attained.

show abstract

“…We use 9 projects that have been widely used in previous software testing research [50], [55], [56] as the cross-project scenario subjects to evaluate PMT. In particular, we use the latest versions (i.e., the HEAD commit) of these projects as the base subjects.…”

Section: Subject Systemsmentioning

confidence: 99%

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

Self Cite

109

View full text Add to dashboard Cite

Abstract-Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60% confidence) when predicting the execution results of the majority of mutants.

show abstract

Operator-based and random mutant selection: Better together

Cited by 74 publications

References 50 publications

An Empirical Study on the Scalability of Selective Mutation Testing

An Empirical Study on the Scalability of Selective Mutation Testing

An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption

Predictive Mutation Testing

Contact Info

Product

Resources

About