Evaluating improvements to a meta-heuristic search for constrained interaction testing

Abstract

Combinatorial interaction testing (CIT) is a cost-effective sampling technique for discovering interaction faults in highly-configurable systems. Constrained CIT extends the technique to situations where some features cannot coexist in a configuration, and is therefore more applicable to real-world software. Recent work on greedy algorithms to build CIT samples now efficiently supports these feature constraints. But when testing a single system configuration is expensive, greedy techniques perform worse than meta-heuristic algorithms, because greedy algorithms generally need larger samples to exercise the same set of interactions. On the other hand, current meta-heuristic algorithms have long run times when feature constraints are present. Neither class of algorithm is suitable when both constraints and the cost of testing configurations are important factors. Therefore, we reformulate one meta-heuristic search algorithm for constructing CIT samples, simulated annealing, to more efficiently incorporate constraints. We identify a set of algorithmic changes and experiment with our modifications on 35 realistic constrained problems and on a set of unconstrained problems from the literature to isolate the factors that improve performance. Our evaluation determines that the optimizations reduce run time by a factor of 90 and accomplish the same coverage objectives with even fewer system configurations. Furthermore, the new version compares favorably with greedy algorithms on real-world problems, and, though our modifications were

Multi-objective test case prioritization in highly configurable systems: A case study

“…pairwise coverage) led to better results than random suites. Based on their finding, our initial population is composed of different orderings of a pairwise test suite generated by the CASA tool [28,29] from the input feature model. …”

“…Given a prioritized test suite, this metric was used to measure how quickly it would detect the faults in Drupal v7.23. For comparative reasons, we measured the APFD values of both, the prioritized suites generated by our adaptation of NSGA-II and the initial pairwise suite generated by the CASA algorithm [28,29] on each execution.…”

A Search-Based Constraint Elicitation in Test Design

“…Otherwise, some test cases would not be executable because of constraint violation, resulting in redoing the test design process. Once we define constraints, we can use test generation tools, such as PICT [6], ACTS [7], and CASA [8], to acquire a set of test cases. These tools accept a test model and constraints as input data and generate the set of test cases.…”

Comparative Benchmarking of Constraints T-Way Test Generation Strategy Based on Late Acceptance Hill Climbing Algorithm

“…In the reported work, SA merely addresses small values of interaction strength(i.e., t≤ 3). Another variant of SA, called CASA (Garvin, Cohen et al 2011), has been developed by to address the support for constraints. Empirical evidences suggest its successful use for software product lines testing.…”