Design and implementation of a t-way test data generation strategy with automated execution tool support

Abstract

a b s t r a c tTo ensure an acceptable level of quality and reliability of a typical software product, it is desirable to test every possible combination of input data under various configurations. However, due to the combinatorial explosion problem, exhaustive testing is practically impossible. Resource constraints, cost factors, and strict time-to-market deadlines are some of the main factors that inhibit such a consideration. Earlier research has suggested that a sampling strategy (i.e., one that is based on a t-way parameter interaction) can be effective. As a result, many helpful t-way sampling strategies have been developed and can be found in the literature.Several advances have been achieved in the last 15 years, which have, in particular, served to facilitate the test planning process by systematically minimizing the test size required (based on certain t-way parameter interactions). Despite this significant progress, the integration and automation of strategies (from planning process to execution) are still lacking. Additionally, strategizing to sample (and construct) a minimum test set from the exhaustive test space is an NP-complete problem; that is, it is often unlikely that an efficient strategy exists that could regularly generate an optimal test set. Motivated by these challenges, this paper discusses the design, implementation, and validation of an efficient strategy for t-way testing, the GTWay strategy. The main contribution of GTWay is the integration of t-way test data generation with automated (concurrent) execution as part of its tool implementation. Unlike most previous methods, GTWay addresses the generation of test data for a high coverage strength (t > 6).

A Review of Covering Arrays and Their Application to Software Testing

“…Wong et al (2010) investigate a new fault localization method using code coverage heuristic. In other recently researches, interaction testing using CAs have been effective for improving code coverage using some empirical studies (Zamli et al, 2011;Klaib et al, 2008). This in turn could be an important motivation for using CAs with code coverage heuristics to improve the fault localization as conducted by Wong et al (2010).…”

A Tabu Search hyper-heuristic strategy for t-way test suite generation

“…Over the years, a number of variations of AETG have emerged including mAETG [27] and mAETG_SAT [34]. Similar to AETG, GTWay [35,36] also adopts the one-test-at-a-time approach to generate the final test suite. Unlike AETG, GTWay permits the use of actual parameter values as a symbolic string and supports automated execution of test cases.…”

Comparative Benchmarking of Constraints T-Way Test Generation Strategy Based on Late Acceptance Hill Climbing Algorithm

“…However, in the case where the number of pairs to be considered is significantly large, adopting computational approaches can be expensive due to the need to consider explicit enumeration from all the combination space. Example strategies that adopt this approach includes An Automatic Efficient Test Generator (AETG) (Cohen, Dalal et al 1996;Cohen, Dalal et al 1997), its variant (mAETG) , PICT [8,36], IPOG (Lei, Kacker et al 2007), Jenny (Pallas 2003), TVG (Yu-Wen and Aldiwan 2000; Arshem 2010), IRPS , GTWay (Klaib 2009;Zamli, Klaib et al 2011), MIPOG Younis and Zamli 2010), TConfig (Williams 2000;Williams 2010), TCG (Tung and Aldiwan 2000)). Another variant of the computational based approach is based on meta-heuristics algorithm including that of GA (Shiba, Tsuchiya et al 2004), HSS (Alsewari and Zamli 2012), PSTG Ahmed and Zamli 2011;Ahmed, Zamli et al 2012), SA (Stardom 2001;Cohen, Colbourn et al 2008), and ACA (Shiba, Tsuchiya et al 2004)).…”

“…In the last 20 years, many t-way strategies have been proposed in literature (including that of GTWay (Klaib 2009;Zamli, Klaib et al 2011), MIPOG Younis and Zamli 2010), TConfig (Williams 2000;Williams 2010), TCG (Tung and Aldiwan 2000), Jenny (Pallas 2003), TVG (Yu-Wen and Aldiwan 2000; Arshem 2010), IRPS , and TConfig (Williams 2000;Williams 2010)). Recently, researchers have also started to adopt meta-heuristics based strategies including that of Genetic Algorithm (GA) GA (Sthamer 1995;Shiba, Tsuchiya et al 2004;Bryce and Colbourn 2007;Afzal, Torkar et al 2009;Chen, Gu et al 2009;McCaffrey 2010), Ant Colony (ACS) ACA (Harman and Jones 2001;Shiba, Tsuchiya et al 2004;Wang, Xu et al 2008;Afzal, Torkar et al 2009;Chen, Gu et al 2009), Particle Swarm Optimization (PSTG) PSTG Ahmed and Zamli 2011;Ahmed, Zamli et al 2012), and Harmony Search Strategy (HSS) (Alsewari and Zamli 2012).…”