Restricted Random Testing: Adaptive Random Testing by Exclusion

Abstract

Restricted Random Testing (RRT) is a new method of testing software that improves upon traditional Random Testing (RT) techniques. Research has indicated that failure patterns (portions of an input domain which, when executed, cause the program to fail or reveal an error) can influence the effectiveness of testing strategies. For certain types of failure patterns, it has been found that a widespread and even distribution of test cases in the input domain can be significantly more effective at detecting failure compared with ordinary RT. Testing methods based on RT, but which aim to achieve even and widespread distributions, have been called Adaptive Random Testing (ART) strategies. One implementation of ART is RRT. RRT uses exclusion zones around executed, but non-failure-causing, test cases to restrict the regions of the input domain from which subsequent test cases may be drawn. In this paper, we introduce the motivation behind RRT, explain the algorithm and detail some empirical analyses carried out to examine the effectiveness of the method. Two versions of RRT are presented: Ordinary RRT (ORRT) and Normalized RRT (NRRT). The two versions share the same fundamental algorithm, but differ in their treatment of non-homogeneous input domains. Investigations into the use of alternative exclusion shapes are outlined, and a simple technique for reducing the computational overheads of RRT, prompted by the alternative exclusion shape investigations, is also explained. The performance of RRT is compared with RT and another ART method based on maximized minimum test case separation (DART), showing excellent improvement over RT and a very favorable comparison with DART.

Distributing test cases more evenly in adaptive random testing

“…F-measure is more appropriate than other testing effectiveness metrics to measure ART/RT, as justified by Chen and Merkel (2008), so we will follow previous studies (Chen et al, 2004b(Chen et al, ,c, 2007cMayer, 2005;Chan et al, 2006) to use F-measure for evaluating the failure-detection capability of ART.…”

“…As it has been reported by Chen et al (2007d) that F ART depends on many factors (such as θ, the dimension of D, the compactness and the number of failure regions, the existence and the size of a predominant failure region), theoretical study of F ART is extremely difficult. Hence, previous studies (Chen et al, 2004b(Chen et al, ,c, 2007cMayer, 2005;Chan et al, 2006) investigated F ART through either simulations or empirical studies. For simulations, θ and the failure pattern were predefined.…”

“…In Restricted Random Testing (RRT) (Chan et al, 2006), an exclusion zone is created around each element of E. The exclusion zone for each element of E is of the same size R · |D| |E| , where R is referred to as the target exclusion ratio. The RRT algorithm continuously generates inputs randomly from D until an input is generated outside all exclusion zones, which is then used as the next test case.…”

“…Based on their work, many ART algorithms have been proposed, such as Fixed-SizedCandidate-Set ART (FSCS-ART) (Chen et al, 2004c), Lattice-based ART (LART) (Mayer, 2005), and Restricted Random Testing (RRT) (Chan et al, 2006). These algorithms have been experimentally evaluated and it was confirmed that ART can use fewer test cases to detect the first failure than RT when failure-causing inputs are clustered into contiguous failure regions.…”

“…Set noT rial = 0 and R = R − 0.1 (if R < 0, then set R = 0 (Chan et al, 2006) have shown that the failuredetection capability of RRT becomes better with the increase of R, but a larger value of R means that a larger part of D would be excluded. Con-sequently, a larger R may result in a longer computation time to find an appropriate test case.…”