Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Abstract

Abstract-Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction.

An Empirical Study of JUnit Test-Suite Reduction

“…There is additional work aiming to improve the fault-detection capability of reduced test suites. For example, Jeffrey et al [17], [30] modified the heuristic algorithm proposed by Harrold et al [2] by selecting redundant test cases to maintain the fault-detection capability.…”

Selecting Highly Efficient Sets of Subdomains for Mutation Adequacy

“…Tallam et al [25] addressed this issue by removing redundant test cases before applying the greedy algorithm. In addition to this, Jeffrey and Gupta [26] use a second set of requirements to determine whether a test case really is redundant. Our solution is to select test cases based on the results of multiple evaluations to avoid overfitting the selection.…”

Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles

“…However, test cases considered as redundant in respect with the main testing criterion may be considered as essential test cases with respect to the additional criteria. This is evidenced by the results of the empirical studies presented in [10]. These results suggest that using several coverage criteria, rather than a single criterion, during test suite reduction is indeed useful in determining test cases which are likely to expose different faults.…”

“…So they will be preserved from discarding because they cover execution paths which may contain some kind of faults. These results are evidenced by empirical studies in [10]. Therefore, using multiple criteria during reduction is useful to select test cases that are more likely to expose faults.…”

“…When a reduction algorithm determines some test cases to be redundant, it uses a certain criterion to do this. But redundant test cases which are thrown away during test suite reduction may exercise unique situations in the software with respect to another criterion [10]. For example, a reduction algorithm may select test cases which cover the most number of branches of a program.…”