Understanding The Impact of Solver Choice in Model-Based Test Generation

Abstract

Background: In model-based test generation, SMT solvers explore the state-space of the model in search of violations of specified properties. If the solver finds that a predicate can be violated, it produces a partial test specification demonstrating the violation. Aims: The choice of solvers is important, as each may produce differing counterexamples. We aim to understand how solver choice impacts the effectiveness of generated test suites at finding faults. Method: We have performed experiments examining the impact of solver choice across multiple dimensions, examining the ability to attain goal satisfaction and fault detection when satisfaction is achieved-varying the source of test goals, data types of model input, and test oracle. Results: The results of our experiment show that solvers vary in their ability to produce counterexamples, and-for models where all solvers achieve goal satisfaction-in the resulting fault detection of the generated test suites. The choice of solver has an impact on the resulting test suite, regardless of the oracle, model structure, or source of testing goals. Conclusions: The results of this study identify factors that impact fault-detection effectiveness, and advice that could improve future approaches to model-based test generation. CCS CONCEPTS• Software and its engineering → Software testing and debugging; System modeling languages.

Symbolic Observation Graph-Based Generation of Test Paths

“…The cornerstone of model-based testing is the generation of test inputs from the behavioural model of the system under test. Several techniques have been introduced for test inputs generation, such as solvers [34], constrained logic programming [32], search-based algorithms [38] and model checking [19]. Except for some educational systems, the number of input data is often infinite.…”