A Comparison of Test Case Prioritization Criteria for Software Product Lines

Abstract

Abstract-Software Product Line (SPL) testing is challenging due to the potentially huge number of derivable products. To alleviate this problem, numerous contributions have been proposed to reduce the number of products to be tested while still having a good coverage. However, not much attention has been paid to the order in which the products are tested. Test case prioritization techniques reorder test cases to meet a certain performance goal. For instance, testers may wish to order their test cases in order to detect faults as soon as possible, which would translate in faster feedback and earlier fault correction. In this paper, we explore the applicability of test case prioritization techniques to SPL testing. We propose five different prioritization criteria based on common metrics of feature models and we compare their effectiveness in increasing the rate of early fault detection, i.e. a measure of how quickly faults are detected. The results show that different orderings of the same SPL suite may lead to significant differences in the rate of early fault detection. They also show that our approach may contribute to accelerate the detection of faults of SPL test suites based on combinatorial testing.

Multi-objective test case prioritization in highly configurable systems: A case study

“…Some pieces of work have shown that two dissimilar test cases have a higher fault detection rate than similar ones since the former ones are more likely to cover more components than the latter [33,60]. This function favors a test suite with the most different test cases in order to cover more features and improve the rate and acceleration of fault detection.…”

“…In our previous work [60], we adapted CoC to HCS configurations achieving good results in accelerating the detection of faults. Now we propose to measure the complexity of features in terms of the number of edges and constraints in which they are involved.…”

“…Functional prioritization objectives are those based on the functional features of the system and their interactions. Some examples are those based on combinatorial interaction testing [75], configuration dissimilarity [2,33,60] or feature model complexity metrics [59,60]. Non-functional prioritization objectives consider extra-functional information such as user preferences [21,39], cost [75], memory consumption [45] or execution probability [15] to find the best ordering for test cases.…”

“…Many pieces of work have used pairwise coverage based on the evidence that a high percentage of detected faults are mainly due to the interactions between two features (e.g. [26,32,60]). This objective function measures and accelerates the pairwise coverage of a test suite, giving priority to those test cases that cover a higher number of pairs of features more quickly.…”

Combinatorial Interaction Testing of Software Product Lines: A Mapping Study

“…Some identified testing techniques are derived from Combinatorial Interaction Testing and Search-Based Testing (Harman et al, 2014), Specification Based Testing (Scheidemann, 2006), Logic Based and various hybrid testing techniques (Perrouin et al, 2010;Hervieu et al, 2011a;Henard et al, 2013;Al-Hajjaji et al, 2014;Sánchez et al, 2014a). Due to the availability of various approaches, it is the motivation of this mapping study to collect and review those works, which leads to the first research question: Details on experiments conducted to the tool and their data sets are gathered in order to get a better understanding on the evaluation aspect of Combinatorial Interaction Testing techniques in SPL.…”