An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing

Abstract

We present an empirical comparison of three test generation techniques, namely, Combinatorial Testing (CT), Random Testing (RT) and Adaptive Random Testing (ART), under different test scenarios. This is the first study in the literature to account for the (more realistic) testing setting in which the tester may not have complete information about the parameters and constraints that pertain to the system, and to account for the challenge posed by faults (in terms of failure rate). Our study was conducted on nine real-world programs under a total of 1683 test scenarios (combinations of available parameter and constraint information and failure rate). The results show significant differences in the techniques' fault detection ability when faults are hard to detect (failure rates are relatively low). CT performs best overall; no worse than any other in 98% of scenarios studied. ART enhances RT, and is comparable to CT in 96% of scenarios, but its computational cost can be up to 3.5 times higher than CT when the program is highly constrained. Additionally, when constraint information is unavailable for a highly-constrained program, a large random test suite is as effective as CT or ART, yet its computational cost of test generation is significantly lower than that of other techniques.

Fault Detection Capabilities of Combinatorial Testing and Random Testing for Boolean-Specifications

“…All the test cases within the input domain have the same probabilities to be selected in random testing. E.g., for a general-form Boolean expression with 5 Boolean input variables, its input domain should contain 2 5 test cases. The probability of the event that an arbitrary test case is picked up in random testing is 1/2 5 .…”

“…Nie et al conducted an experiment on nine programs, including FLEX, GREP, GZIP, SED, MAKE, NANOXML, DRUPAL, BUSYBOX, and LINUX. Their results suggested that combinatorial testing is better than random testing in many cases, especially when the failure rate is lower [5].…”

“…Many people designed experiments to answer the question of whether the combinatorial test suite has a higher fault detection capacity than the random test suite containing the same number of test cases. Some studies reported positive results that combinatorial testing was more effective than random testing [2][3][4][5], while some other studies reported negative results [6][7]. Most of the studies that gave positive results focused on Boolean-specification testing [2][3][4].…”

Comparative Analysis of Constraint Handling Techniques for Constrained Combinatorial Testing

“…A test case of the SUT is then produced by assigning to each parameter a specific value [1]. Table 1, for example, shows a test model for testing font effects in a word processor (this example is adapted from previous work [29]). This test model has five parameters: Style and Underline Colour can take three values, and each of the others can take two values.…”

A Comparison of Fault Detection Efficiency Between Adaptive Random Testing and Greedy Combinatorial Testing for Control Logics in Nuclear Industrial Distributed Control Systems

“…In the empirical studies on combinatorial testing, people usually focus on fault detection ability rather than fault detection efficiency [7]. People have compared the fault detection capabilities of adaptive random testing and combinatorial testing in empirical studies [8], but the comparison of the fault detection efficiencies of them seems to be lacking. When testing a nuclear industrial DCS, the cost of each test case's execution is high since it involves both software and hardware.…”