Prioritization of Combinatorial Test Cases by Incremental Interaction Coverage

Abstract

Combinatorial interaction testing is a well-recognized testing method, and has been widely applied in practice, often with the assumption that all test cases in a combinatorial test suite have the same fault detection capability. However, when testing resources are limited, an alternative assumption may be that some test cases are more likely to reveal failure, thus making the order of executing the test cases critical. To improve testing cost-e®ectiveness, prioritization of combinatorial test cases is employed. The most popular approach is based on interaction coverage, which prioritizes combinatorial test cases by repeatedly choosing an unexecuted test case that covers the largest number of uncovered parameter value combinations of a given strength (level of interaction among parameters). However, this approach su®ers from some drawbacks. Based on previous observations that the majority of faults in practical systems can usually be triggered with parameter interactions of small strengths, we propose a new strategy of prioritizing combinatorial test cases by incrementally adjusting the strength values. Experimental results show that our method performs better than the random prioritization technique and the technique of prioritizing combinatorial test suites according to test case . Downloaded from www.worldscientific.com by UNIVERSITY OF OTAGO on 01/05/15. For personal use only.generation order, and has better performance than the interaction-coverage-based test prioritization technique in most cases.

Combinatorial Test Case Prioritization based on Incremental Combinatorial Coverage Strength

“…This paper could be considered an extended version of that. Huang et al did the same work [12]. The technique based on adaptive random strategy [10] and the technique for aggregate-strength [11] were proposed in recent years.…”

Adaptive random prioritization for interaction test suites

“…Besides, Huang et al [6] proposed two heuristic methods to prioritize a special CA named variable-strength covering array or VCA, which is also available to be used to guide the CA prioritization. Additionally, Huang et al [8] proposed a new method based on incremental interaction coverage in order to prioritize the arbitrary interaction test suites, which is not limited to CAs or VCAs. In other words, their method can also be used to be successfully applied to CAs or VCAs.…”

New Metrics for Prioritized Interaction Test Suites

“…Reference Pure prioritization [2], [5], [6], [8], [9], [10], [13] Re-generation prioritization [2], [3], [4], [7], [9], [11], [12] require 3 5 = 243 test cases. However, as shown in Table 2, a covering array CA (12, 2, 5, 3 5 ) requires only 12 test cases to cover all 2-wise value combinations.…”

“…The size of each covering array is given in Table 4. Additionally, to obtain different interaction test sequences, we used four prioritization strategies that have commonly used in the field of combinatorial interaction testing -(1) Original: generating the interaction test sequence according to the test case order of the corresponding interaction test suite; (2) Random: prioritizing the interaction test suite in a random manner [2]; (3) ICBP: prioritizing the interaction test suite based on interaction coverage of a given strength, namely, Interaction Coverage Based Prioritization (ICBP) [2], [5], [6], [8], [9]; and (4) IICBP: prioritizing the interaction test suite based on different interaction coverage by incrementally updating the strength and starting with a strength of 1, namely, Incremental Interaction Coverage Based Prioritization (IICBP) [13].…”

“…To date, many strategies have been proposed to guide the prioritization of interaction test suites [2]- [13], including branch-coverage-based test prioritization [2] and random test case prioritization [8]. Given an interaction test suite, different interaction test sequences can be obtained by using different test prioritization strategies.…”