Minimizing test suites in software product lines using weight-based genetic algorithms

Abstract

Test minimization techniques aim at identifying and eliminating redundant test cases from test suites in order to reduce the total number of test cases to execute, thereby improving the efficiency of testing. In the context of software product line, we can save effort and cost in the selection and minimization of test cases for testing a specific product by modeling the product line. However, minimizing the test suite for a product requires addressing two potential issues: 1) the minimized test suite may not cover all test requirements compared with the original suite; 2) the minimized test suite may have less fault revealing capability than the original suite. In this paper, we apply weight-based Genetic Algorithms (GAs) to minimize the test suite for testing a product, while preserving fault detection capability and testing coverage of the original test suite. The challenge behind is to define an appropriate fitness function, which is able to preserve the coverage of complex testing criteria (e.g., Combinatorial Interaction Testing criterion). Based on the defined fitness function, we have empirically evaluated three different weightbased GAs on an industrial case study provided by Cisco Systems, Inc. Norway. We also presented our results of applying the three weight-based GAs on five existing case studies from the literature. Based on these case studies, we conclude that among the three weight-based GAs, Random-Weighted GA (RWGA) achieved significantly better performance than the other ones.

Variability testing in the wild: the Drupal case study

“…Related work on variability testing mainly addresses the problems of test case selection [4,6,16,21,32,37,39,40,42,48,55,66] and test case prioritization [2,6,15,20,22,31,35,45,54,61,67]. Most approaches use functional information to drive testing such as those based on combinatorial testing [2,22,31,32,35,37,39,40,42,48,54,67,61,66], similarity [2,31,54] or other metrics extracted from the feature model [16,21,32,54,55].…”

“…Most approaches use functional information to drive testing such as those based on combinatorial testing [2,22,31,32,35,37,39,40,42,48,54,67,61,66], similarity [2,31,54] or other metrics extracted from the feature model [16,21,32,54,55]. Several works have also explored the use of non-functional properties during testing such as user preferences and cost [6,14,15,20,22,23,32,35,55,61,66,67]. The lack of realistic case studies often lead researchers to evaluate their approaches using synthetic variability models [2,4,20,21,22,32,37,39,54,55], faults [2,15,16,…”

Multi-objective test case prioritization in highly configurable systems: A case study

“…suite size) but not both at the same time. Other works [70,75] combine several objectives into a single function by assigning them weights proportional to their relative importance. While this may be acceptable in certain scenarios, it may be unrealistic in others where users may wish to study the trade-offs among several objectives [44].…”

“…The work by Wang et al describe an approach to minimize test suites using three objectives [70], namely, test minimization percentage, pairwise coverage, and fault detection capability that works by assigning weights to these objectives -a process called scalarization [82]. Their work was extended to generate weights from a uniform distribution while still satisfying the userdefined constraints [71].…”

A Univariate Marginal Approach for Pairwise Testing of Software Product Lines

“…Wang et al [14] use a weighted Genetic Algorithm to minimize SPL test suites, and at the same time maintain fault detecting power. Haslinger et al [15] applied a Simulated Annealing algorithm to generate t-wise covering array and demonstrated a tool to improve the performance of SPL testing.…”