The impact of test suite granularity on the cost-effectiveness of regression testing

Abstract

Regression testing is an expensive testing process used to validate software following modi cations. The cost-e ectiveness of regression testing techniques varies with characteristics of test suites. One such characteristic, test suite granularity, involves the way in which test inputs are grouped into test cases within a test suite. Various cost-bene ts tradeo s have been attributed to choices of test suite granularity, but almost no research has formally examined these tradeo s.To address this lack, we conducted several controlled experiments, examining the e ects of test suite granularity on the costs and bene ts of several regression testing methodologies across six releases of two non-trivial software systems. Our results expose essential tradeo s to consider when designing test suites for use in regression testing evolving systems.

Reducing the Costs of Bounded-Exhaustive Testing

“…In bounded-exhaustive testing, users typically describe a test set with a large number of small tests, while we advocate considering test sets with a smaller number of larger tests. Our technique is inspired by the work on test granularity [18,19] which studied the cost-benefit trade-offs in using a larger number of smaller tests versus a smaller number of larger tests. That work mostly considered manually written tests for regression testing, while we focus on automatically generated tests.…”

“…We refer to the number of While STM achieves significant time savings, it is important to note its two potential drawbacks. One potential drawback is that larger inputs, through the interference of program elements, can mask some test failures [18,19]. Consider, for example, merging together all the different field references (as in Figure 4).…”

“…STM is related to work on test granularity [18,19] which studied the costbenefit trade-offs in testing with a larger number of smaller tests versus a smaller number of larger tests. The key difference is that previous work considered tests that can be easily appended while we consider tests that need to be merged.…”

Empirical Studies of Test Case Prioritization in a JUnit Testing Environment

“…Recent studies of prioritization have begun to examine the factors affecting prioritization effectiveness [11,22,25]. Rothermel et al [25] studied the effects of test suite design on regression testing techniques, varying the composition of test suites and examining the effects on cost-effectiveness of test selection and prioritization.…”

“…Rothermel et al [25] studied the effects of test suite design on regression testing techniques, varying the composition of test suites and examining the effects on cost-effectiveness of test selection and prioritization. While this study did not consider correlating attributes of change with technique performance, Elbaum et al [11] performed experiments exploring characteristics of program structure, test suite composition, and changes on prioritization, and identified several metrics characterizing these attributes that correlate with prioritization effectiveness.…”