Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines

Abstract

Large Software Product Lines (SPLs) are common in industry, thus introducing the need of practical solutions to test them. To this end, t-wise can help to drastically reduce the number of product configurations to test. Current t-wise approaches for SPLs are restricted to small values of t. In addition, these techniques fail at providing means to finely control the configuration process. In view of this, means for automatically generating and prioritizing product configurations for large SPLs are required. This paper proposes (a) a search-based approach capable of generating product configurations for large SPLs, forming a scalable and flexible alternative to current techniques and (b) prioritization algorithms for any set of product configurations. Both these techniques employ a similarity heuristic. The ability of the proposed techniques is assessed in an empirical study through a comparison with state of the art tools. The comparison focuses on both the product configuration generation and the prioritization aspects. The results demonstrate that existing t-wise tools and prioritization techniques fail to handle large SPLs. On the contrary, the proposed techniques are both effective and scalable. Additionally, the experiments show that the similarity heuristic can be used as a viable alternative to t-wise.

Combinatorial Interaction Testing of Software Product Lines: A Mapping Study

“…Over ninety percent of the reviewed primary studies were engaging t-wise of strength two or three, further substantiating claims made by Henard et al (2014). The main limitation that is causing this phenomenon is due to the expensive computation time required to deploy higher strength twise testing.…”

“…The results on the type and size of data sets utilized by the selected primary studies suggests that not only lower strength of t-wise is being handled by majority of current works, as what being highlighted by Henard et al (2014), but it is generally limited to small and medium sized SPL systems.…”

“…Since most of current works are focusing on lower strength (less than 4) of t-wise combinations of features (Henard et al, 2014), it is interesting to ask whether this phenomenon has any co-relation with the size of the data sets that were utilized in the experiments.…”

A Combination Test Suite Generation Method Based on Adaptive Simulated Annealing Genetic Algorithm for Software Product Line Testing

“…And many combination testing methods have been applied to SPL feature models so far, e.g. [5] [6]. Considering search-based methods to search a good test suite in the search space of the valid set which is generated from SPL feature models and constraints between features, this paper shows SPL feature model definition as formula (1).…”

“…Proposed in the reference, e.g. [5], which gives a optimization method to generate a combination test suite for SPL testing, the method can generate a test suite with higher coverage compared with the random search method. But the condition that the variability equal to 1 is to be constant is adverse to the searching efficiency in the process of composition optimization.…”