Model-Based Filtering of Combinatorial Test Suites

Abstract

Abstract.Tobias is a combinatorial test generation tool which can efficiently generate a large number of test cases by unfolding a test pattern and computing all combinations of parameters. In this paper, we first propose a model-based testing approach where Tobias test cases are first run on an executable UML/OCL specification. This animation of test cases on a model allows to filter out invalid test sequences produced by blind enumeration, typically the ones which violate the pre-conditions of operations, and to provide an oracle for the valid ones. We then introduce recent extensions of the Tobias tool which support an incremental unfolding and filtering process, and its associated toolset. This allows to address explosive test patterns featuring a large number of invalid test cases, and only a small number of valid ones. For instance, these new constructs could mandate test cases to satisfy a given predicate at some point or to follow a given behavior. The early detection of invalid test cases improves the calculation time of the whole generation and execution process, and helps fighting combinatorial explosion.

Test suite selection based on traceability annotations

“…Other techniques to master combinatorial explosion require to execute test cases [9], or to simulate them on a model [12]. These techniques are very efficient to rule out of the resulting test suite a large number of test cases which fail to meet predefined conditions or behaviours.…”

Test Generation and Evaluation from High-Level Properties for Common Criteria Evaluations -- The TASCCC Testing Tool

“…For example, a robustness scenario created from Property 1 counts 1.89 billion test cases and it would take too much time to animate every test case. Therefore, an incremental evaluation algorithm has been used to discard all test cases corresponding to the same prefix as soon as this prefix fails [3]. Using this incremental evaluation, the tool can select the 126 valid test cases out of 1.89 billion candidates, in less than 4 minutes on a standard PC.…”

“…Such scenarios then need to be unfolded and filtered so as to eliminate sequences that would not be executable at the model level. To do so, two complementary tools interact: TOBIAS [3], which is in charge of unfolding the scenario into (unverified) test cases, and the model animation engine of CertifyIt, which is in charge of animating each test case ). This test suite can then be concretized into test scripts to be run on the system under test.…”