Practical Combinatorial Interaction Testing: Empirical Findings on Efficiency and Early Fault Detection

Abstract

Combinatorial Interaction Testing (CIT) is important because it tests the interactions between the many features and parameters that make up the configuration space of software systems. Simulated Annealing (SA) and Greedy Algorithms have been widely used to find CIT test suites. From the literature, there is a widely-held belief that SA is slower, but produces more effective tests suites than Greedy and that SA cannot scale to higher strength coverage. We evaluated both algorithms on 7 real-world subjects for the well-studied 2-way up to the rarely-studied 6-way interaction strengths. Our findings present evidence to challenge this current orthodoxy: real-world constraints allow SA to achieve higher strengths. Furthermore, there was no evidence that Greedy was less effective (in terms of time to fault revelation) compared to SA; the results for the greedy algorithm are actually slightly superior. However, the results are critically dependent on the approach adopted to constraint handling. Moreover, we have also evaluated a genetic algorithm for constrained CIT test suite generation. This is the first time strengths higher than 3 and constraint handling have been used to evaluate GA. Our results show that GA is competitive only for pairwise testing for subjects with a small number of constraints.

Validation of Constraints Among Configuration Parameters Using Search-Based Combinatorial Interaction Testing

“…There exist several tools and techniques for CIT. Good surveys of ongoing research in CIT can be found in [9,19], while an introduction to CIT and its efficacy in practice can be found in [15,21].…”

“…It is a very popular black-box testing technique that tests all interactions between any set of t parameters. There have been several studies showing the successful efficacy and efficiency of the approach [13,14,21].…”

“…Designers, developers, and testers can greatly benefit from modelling parameters and constraints among them by significantly reducing modelling and testing effort [21] as well as identifying corner cases of the system under test. Constraints play a very important role, since they identify parameter interactions that need not be tested, hence they can significantly reduce the testing effort.…”

Test Effectiveness Evaluation of Prioritized Combinatorial Testing: A Case Study

“…To evaluate the efficiency of combinatorial t-way testing, Petke et al [15] investigate fault detection effectiveness of combinatorial t-way test suites (2 ≤ t ≤ 6) that are generated by a simulated annealing algorithm, CASA [19], and a greedy algorithm, ACTS [18]. They also examine the fault detection rate of test prioritization of the t-way test suites w. r. t. t -way interaction coverage with 2 ≤ t ≤ 6, which means the test suites whose test cases are re-ordered in the descent order of t -way coverage.…”

“…While we in this paper explore weight coverage and KL divergence with fault detection effectiveness of prioritized combinatorial testing with weighted SUT, the work [15], [7], [8] consider combinatorial testing with non-weighted SUT and investigate neither weight coverage nor KL divergence.…”