Balancing trade-offs in test-suite reduction

Shi, August; Gyori, Alex; Gligoric, Milos; Zaytsev, Andrey; Marinov, Darko

doi:10.1145/2635868.2635921

Cited by 82 publications

(42 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mutation testing tools. We use two popular Java mutation tools: PIT 3 [49] and Major 4 , both of which have been widely used in mutation testing research [50], [51]. As our work targets the cost problem of mutation testing, we choose PIT as the primary mutation testing tool since it is evaluated to be very efficient [52].…”

Section: Propagationmentioning

confidence: 99%

“…We use 9 projects that have been widely used in previous software testing research [50], [55], [56] as the cross-project scenario subjects to evaluate PMT. In particular, we use the latest versions (i.e., the HEAD commit) of these projects as the base subjects.…”

Section: Subject Systemsmentioning

confidence: 99%

“…In particular, we use the latest versions (i.e., the HEAD commit) of these projects as the base subjects. To facilitate the evaluation of PMT in the cross-version scenario, we prepare multiple versions for each base subject as in existing work [50]. More specifically, we select each version by counting backwards 30 commits at a time from the latest version commit of each project and generate up to 10 versions per project.…”

Section: Subject Systemsmentioning

confidence: 99%

“…To reduce the threat resulting from subjects, we selected our base project versions following the same procedure as that used in the prior work [50] and used a large number of Java projects to cater for the project variability. To reduce the threat of mutation tools, we used both PIT and Major in our study because the former is efficient The correlation between predictability and killability of mutants.…”

Section: Threats To Validitymentioning

confidence: 99%

See 3 more Smart Citations

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

109

View full text Add to dashboard Cite

Abstract-Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60% confidence) when predicting the execution results of the majority of mutants.

show abstract

Section: Propagationmentioning

confidence: 99%

Section: Subject Systemsmentioning

confidence: 99%

Section: Subject Systemsmentioning

confidence: 99%

Section: Threats To Validitymentioning

confidence: 99%

See 2 more Smart Citations

Predictive Mutation Testing

Zhang

Harman

et al. 2019

IIEEE Trans. Software Eng.

109

View full text Add to dashboard Cite

show abstract

“…For problems concerned with test suite selection and prioritisation, multi-objective approaches are increasingly prevalent [8], [13], [17], [87], [105], [106], [125]. However, for test data generation problems, the large majority of existing approaches are single objective.…”

Section: Multi-objective Search Based Testing (Mosbat)mentioning

confidence: 99%

Achievements, Open Problems and Challenges for Search Based Software Testing

Harman¹,

Jia²,

Zhang³

2015

2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST)

157

View full text Add to dashboard Cite

Abstract-Search Based Software Testing (SBST) formulates testing as an optimisation problem, which can be attacked using computational search techniques from the field of Search Based Software Engineering (SBSE). We present an analysis of the SBST research agenda 1 , focusing on the open problems and challenges of testing non-functional properties, in particular a topic we call 'Search Based Energy Testing' (SBET), Multi-objective SBST and SBST for Test Strategy Identification. We conclude with a vision of FIFIVERIFY tools, which would automatically find faults, fix them and verify the fixes. We explain why we think such FIFIVERIFY tools constitute an exciting challenge for the SBSE community that already could be within its reach.

show abstract

Analysis and assessment of weighted combinatorial criterion for test suite reduction

Chiang

Huang

Chiu

et al. 2021

Quality & Reliability Eng

View full text Add to dashboard Cite

Software plays an important role in modern society. New software features have to be completed and tested rapidly due to business and user needs and goals. To ensure the quality of modified software components, new test cases must be generated and included in the current test pool. Furthermore, test pool size often grows too large to test all cases. A considerably large test pool typically consumes an exorbitant quantity of time, leading to inefficient regression testing. Test suite reduction is often used to solve this problem by removing redundant test cases. Thus, time and resource constraints make it necessary to minimize the test suite. That is, test pool size should be efficiently reduced, while the remaining test cases should be adequate to provide the same coverage as the pool. The main purpose of this paper is to propose four weighted combinatorial algorithms and a three‐objective binary integer linear programming (ILP) model to improve fault detection effectiveness. We first consider both the control flow and data flow as our testing criteria and combine them by using a weighted means with four existing test suite reduction algorithms. A genetic algorithm is used to find the best combination of weighting factors to assign to the two testing criteria. Experiments based on real subject programs are presented. Experimental results demonstrate that 77.5% of our proposed weighted combinatorial algorithms have an even or better performance in suite size reduction, 83.75% of our approaches perform even or better in fault detection effectiveness, and 82.5% of our approaches have even or better effectiveness in fault‐to‐test ratio. For the three‐objective binary ILP model, test suite size is significantly reduced without any accompanying loss in fault detection ability.

show abstract

Balancing trade-offs in test-suite reduction

Cited by 82 publications

References 44 publications

Predictive Mutation Testing

Predictive Mutation Testing

Achievements, Open Problems and Challenges for Search Based Software Testing

Analysis and assessment of weighted combinatorial criterion for test suite reduction

Contact Info

Product

Resources

About