Introduction to Combinatorial Testing

Kuhn, D. Richard; Kacker, Raghu N.; Lei, Yu

doi:10.1201/b14957

Cited by 87 publications

(119 citation statements)

References 0 publications

Supporting

Mentioning

119

Contrasting

Order By: Relevance

“…The sampling heuristics and experimental designs we use to identify interactions are related to the heuristics used in combinatorial testing [10,15]. The difference is that we do not focus on functional correctness, but on performance, which allows us to learn performance-influence models using linear regression.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Performance-influence models for highly configurable systems

Siegmund

Grebhahn

Apel

et al. 2015

Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

200

245

View full text Add to dashboard Cite

Almost every complex software system today is configurable. While configurability has many benefits, it challenges performance prediction, optimization, and debugging. Often, the influences of individual configuration options on performance are unknown. Worse, configuration options may interact, giving rise to a configuration space of possibly exponential size. Addressing this challenge, we propose an approach that derives a performance-influence model for a given configurable system, describing all relevant influences of configuration options and their interactions. Our approach combines machine-learning and sampling heuristics in a novel way. It improves over standard techniques in that it (1) represents influences of options and their interactions explicitly (which eases debugging), (2) smoothly integrates binary and numeric configuration options for the first time, (3) incorporates domain knowledge, if available (which eases learning and increases accuracy), (4) considers complex constraints among options, and (5) systematically reduces the solution space to a tractable size. A series of experiments demonstrates the feasibility of our approach in terms of the accuracy of the models learned as well as the accuracy of the performance predictions one can make with them.

show abstract

Section: Related Workmentioning

confidence: 99%

“…A Plackett-Burman design is a specific type of fractional factorial design, which are often used for combinatorial testing [15]. The design specifies seeds depending on the number of experiments to be conducted and the number of levels of the input variables.…”

Section: Numeric-option Samplingmentioning

confidence: 99%

Performance-influence models for highly configurable systems

Siegmund

Grebhahn

Apel

et al. 2015

Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

200

245

View full text Add to dashboard Cite

show abstract

“…Chen and Zhang [9] proposed a metric for t-way testing called tuple density, which is defined as the (t + 1)-way coverage plus t. Metrics of t (> t)-way coverage of t-way test suites, like tuple density, are important [28], [29], [37] because they distinguish between two t-way test suites with the same t-way coverage from the viewpoint of higher interaction strengths.…”

Section: B T-way Testing With Higher T (> T)-way Coveragementioning

confidence: 99%

“…Combinatorial t-way testing [28], [34]-here t is a small number called the interaction strength-is a well-known black-box testing technique based on a coverage criterion called t-way coverage, which measures how many of the all possible interactions of t parameters are tested. Based on the observation that most system failures are caused by only a few parameters [30], [42], t-way testing aims at ensuring the quality of software testing by stipulating to test all t-way parameter interactions at least once.…”

Section: Introductionmentioning

confidence: 99%

Distance-Integrated Combinatorial Testing

Choi

Artho

Kitamura

et al. 2016

2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)

View full text Add to dashboard Cite

Abstract-This paper proposes a novel approach to combinatorial test generation, which achieves an increase of not only the number of new combinations but also the distance between test cases. We applied our distance-integrated approach to a stateof-the-art greedy algorithm for traditional combinatorial test generation by using two distance metrics, Hamming distance, and a modified chi-square distance. Experimental results using numerous benchmark models show that combinatorial test suites generated by our approach using both distance metrics can improve interaction coverage for higher interaction strengths with low computational overhead.

show abstract

“…[24]) aims at ensuring the quality of software testing by focusing on the interactions of parameters in a system under test (SUT), while at the same time reducing the number of test cases that has to be executed. It has been shown empirically [23] that a significant number of defects can be detected by t-way testing, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

Section: Introductionmentioning

confidence: 99%

Greedy combinatorial test case generation using unsatisfiable cores

Yamada

Biere

Artho

et al. 2016

Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Combinatorial testing aims at covering the interactions of parameters in a system under test, while some combinations may be forbidden by given constraints (forbidden tuples).In this paper, we illustrate that such forbidden tuples correspond to unsatisfiable cores, a widely understood notion in the SAT solving community. Based on this observation, we propose a technique to detect forbidden tuples lazily during a greedy test case generation, which significantly reduces the number of required SAT solving calls. We further reduce the amount of time spent in SAT solving by essentially ignoring constraints while constructing each test case, but then "amending" it to obtain a test case that satisfies the constraints, again using unsatisfiable cores. Finally, to complement a disturbance due to ignoring constraints, we implement an efficient approximative SAT checking function in the SAT solver Lingeling.Through experiments we verify that our approach significantly improves the efficiency of constraint handling in our greedy combinatorial testing algorithm.

show abstract

Introduction to Combinatorial Testing

Cited by 87 publications

References 0 publications

Performance-influence models for highly configurable systems

Performance-influence models for highly configurable systems

Distance-Integrated Combinatorial Testing

Greedy combinatorial test case generation using unsatisfiable cores

Contact Info

Product

Resources

About