Diversity of graph models and graph generators in mutation testing

Abstract

When custom modeling tools are used for designing complex safety-critical systems (e.g., critical cyber-physical systems), the tools themselves need to be validated by systematic testing to prevent tool-specific bugs reaching the system. Testing of such modeling tools relies upon an automatically generated set of models as a test suite. While many software testing practices recommend that this test suite should be diverse, model diversity has not been studied systematically for graph models. In the paper, we propose different diversity metrics for models by generalizing and exploiting neighborhood and predicate shapes as abstraction. We evaluate such shape-based diversity metrics using various distance functions in the context of mutation testing of graph constraints and access policies for two separate industrial DSLs. Furthermore, we evaluate the quality (i.e., bug detection capability) of different (random and consistent) model generation techniques for mutation testing purposes.

Automated Generation of Consistent Graph Models With Multiplicity Reasoning

“…• We evaluate the effectiveness of the approach on numerous case studies including a running example of a complex design space exploration challenge [33] introduced by researchers at NASA Jet Propulsion Lab. The current paper builds upon but substantially extends past research results in [8], [20], [32]. More specifically, the introduction and handling scope constraints are novel conceptual results of the current paper.…”

“…There is a wide range of model generators such as Alloy [12], [13], Formula [14], [15], USE [16], UML2CSP [17], SDG [18], [19] and Viatra Solver [20], [21] to automatically derive consistent models for a given domain specification. Several generators are based on precise foundations offered by backend logic solvers (like SAT solvers [22], [23] or SMT solvers [24]).…”

“…These tools excel at finding inconsistencies (if they exist) by interpreting domain specifications as a logic problem, but they can only derive small consistent models. Moreover, they fail to derive a diverse set of models [20], [25], which restricts their use in practical testing scenarios.…”

“…In order to maintain the favorable theoretical properties (e.g. completeness, diversity) of the generic model generation framework formally proved in [8], [20], the refinement calculus is extended here to incorporate scopes. The prototype implementation builds on and extends [21], [32] by integrating various numerical solvers into the decision procedure.…”

Automated Generation of Consistent Graph Models with First-Order Logic Theorem Provers

“…External Validity: Our approach is limited to a single domain selected based on its past use in related lines of existing research [27,29,30,37]. The domain of Yakindu Statecharts is sufficiently complex to cover all features of our mapping, thus we expect similar scalability results in other domains.…”