Greedy combinatorial test case generation using unsatisfiable cores

Yamada, Akihisa; Biere, Armin; Artho, Cyrille; Kitamura, Takashi; Choi, Eun-Hye

doi:10.1145/2970276.2970335

Cited by 39 publications

(29 citation statements)

References 39 publications

(51 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RELATED WORK One of the closest work to ours is [4]; however, the aforementioned approach 1) operates on partial configurations, which necessitates many more invocations of a solver; 2) depends on a particular covering array constructor, thus requires an adaptation for different constructors; and 3) determines all the valid t-tuples only after a valid covering array is computed. In another work [7], the authors check the list of t-tuples one by one. While doing that, if a new forbidden tuple is discovered, the t-tuples containing this newly found forbidden tuple are removed from the list.…”

Section: Resultsmentioning

confidence: 99%

Enumerator: An Efficient Approach for Enumerating all Valid t-tuples

Mercan

Kaya

Yılmaz

2018

2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

View full text Add to dashboard Cite

show abstract

Section: Resultsmentioning

confidence: 99%

Enumerator: An Efficient Approach for Enumerating all Valid t-tuples

Mercan

Kaya

Yılmaz

2018

2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

View full text Add to dashboard Cite

show abstract

“…A recent SAT-based approach (Yamada et al 2016), implemented in the Calot tool, performed well in terms of efficiency (time to generate the test suites) and cost (test suite sizes) comparing again with the greedy tools ACTS ) and PICT (Czerwonka 2006). Despite the advantages of the SAT-based approach, ACTS was much more faster than Calot for many 3-way test case examples.…”

Section: Introductionmentioning

confidence: 99%

“…CIT approaches to generate test cases can be divided in four main classes: Binary Decision Diagrams (BDDs) (Segall et al 2011), Satisfiability (SAT) solving (Cohen et al 1997;Yamada et al 2015;Yamada et al 2016), meta-heuristics (Garvin et al 2011;Shiba et al 2004;Hernandez et al 2010), and greedy algorithms (Lei and Tai 1998;Lei et al 2007) 1 . Recent CIT test case generation methods based on BDD and SAT are interesting to constrained (there are restrictions related to parameter interactions) problems but they perform worse compared with greedy algorithms/tools in the context of unconstrained (there are no restrictions at all) problems.…”

Section: Introductionmentioning

confidence: 99%

“…Although we have some recent rigorous empirical evaluations comparing greedy algorithms with meta-heuristics solutions (Petke et al 2015) and greedy approaches against SAT-based methods (Yamada et al 2016), there are no rigorous empirical assessments comparing greedy algorithms/tools, representative of the unconstrained CIT domain, among each other.…”

Section: Introductionmentioning

confidence: 99%

“…In case of constrained CIT, constraints must be considered and other definitions can be used (see e.g. (Yamada et al 2016)). …”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Balera

Júnior

2017

J Softw Eng Res Dev

View full text Add to dashboard Cite

Background: Combinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in detecting faults due to the interaction of several input parameters. Recent empirical studies show that greedy algorithms are still competitive for CIT. It is thus interesting to investigate new approaches to address CIT test case generation via greedy solutions and to perform rigorous evaluations within the greedy context. Methods: We present a new greedy algorithm for unconstrained CIT, T-Tuple Reallocation (TTR), to generate CIT test suites specifically via the Mixed-value Covering Array (MCA) technique. The main reasoning behind TTR is to generate an MCA M by creating and reallocating t-tuples into this matrix M, considering a variable called goal (ζ ). We performed two controlled experiments addressing cost-efficiency and only cost. Considering both experiments, we did 3200 executions related to 8 solutions. In the first controlled experiment, we compared versions 1.1 and 1.2 of TTR in order to check whether there is significant difference between both versions of our algorithm. In such experiment, we jointly considered cost (size of test suites) and efficiency (time to generate the test suites) in a multi-objective perspective. In the second controlled experiment we confronted TTR 1.2 with five other greedy algorithms/tools for unconstrained CIT: IPOG-F, jenny, IPO-TConfig, PICT, and ACTS. We performed two different evaluations within this second experiment where in the first one we addressed cost-efficiency (multi-objective) and in the second only cost (single objective). Results: Results of the first controlled experiment indicate that TTR 1.2 is more adequate than TTR 1.1 especially for higher strengths (5, 6). In the second controlled experiment, TTR 1.2 also presents better performance for higher strengths (5, 6) where only in one case it is not superior (in the comparison with IPOG-F). We can explain this better performance of TTR 1.2 due to the fact that it no longer generates, at the beginning, the matrix of t-tuples but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M. Conclusion: Considering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). For lower strengths, other solutions, like IPOG-F, may be better alternatives.

show abstract

Balanced covering arrays: A classification of covering arrays and packing arrays via exact methods

Kampel

Hiess

Kotsireas

et al. 2023

J of Combinatorial Designs

View full text Add to dashboard Cite

In this paper we investigate the intersections of classes of covering arrays (CAs) and packing arrays (PAs). The arrays appearing in these intersections obey to upper and lower bounds regarding the appearance of tuples in sub‐matrices—we call these arrays balanced covering arrays. We formulate and formalize first observations for which upper and lower bounds on the appearance of tuples it is of interest to consider these intersections of CAs and PAs. Outside of these bounds the intersections will be either empty, for the case of too restrictive constraints, or equal to the maximum element in the emerging lattices, for the case of too weak constraints. We present a column extension algorithm for classification of nonequivalent balanced CAs that uses a SAT solver or a pseudo‐Boolean (PB) solver to compute the columns suitable for array extension together with a lex‐leader ordering to identify unique representatives for each equivalence class of balanced CAs. These computations bring to light a dissection of classes of CAs that is partially nested due to the nature of the considered intersections. These dissections can be trivial, containing only a single type of balanced CAs, or can also appear as highly structured containing multiple nested types of balanced CAs. Our results indicate that balanced CAs are an interesting class of designs that is rich of structure.

show abstract

Greedy combinatorial test case generation using unsatisfiable cores

Cited by 39 publications

References 39 publications

Enumerator: An Efficient Approach for Enumerating all Valid t-tuples

Enumerator: An Efficient Approach for Enumerating all Valid t-tuples

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Balanced covering arrays: A classification of covering arrays and packing arrays via exact methods

Contact Info

Product

Resources

About