The Train Benchmark: cross-technology performance evaluation of continuous model queries

Abstract

In model-driven development of safety-critical systems (like automotive, avionics or railways), well-formedness of models is repeatedly validated in order to detect design flaws as early as possible. In many industrial tools, validation rules are still often implemented by a large amount of imperative model traversal code which makes those rule implementations complicated and hard to maintain. Additionally, as models are rapidly increasing in size and complexity, efficient execution of validation rules is challenging for the currently available tools. Checking well-formedness constraints can be captured by declarative queries over graph models, while model update operations can be specified as model transformations. This paper presents a benchmark for systematically assessing the scalability of validating and revalidating well-formedness constraints over large graph models. The benchmark defines well-formedness validation scenarios in the railway domain: a metamodel, an instance model generator and a set of well-formedness constraints captured by queries, fault injection and repair operations (imitating the work of systems engineers by model transformations). The benchmark focuses on the performance of query evaluation, i.e. its execution time and memory consumption, with a particular emphasis on reevaluation. We demonstrate that the benchmark can be adopted to various technologies and query engines, including modeling tools; relational, graph and semantic databases. The Train Benchmark is available as an open-source project with continuous builds from https://github.com/FTSRG/trainbenchmark.

Evaluation of Optimization Strategies for Incremental Graph Queries

“…We use the Train Benchmark [33] as a running example to present both the concepts used in the paper and also as a benchmark framework for evaluating our results. The benchmark defines a domain model for railway networks-the railway domain itself reflects that model validation scenarios are typical in design tools of critical systems.…”

“…Here, we only elaborate the two constraints used in the paper. The full set of constraints and transformations of the Train Benchmark are described in [33].…”

“…For the measurements, we used the open-source Train Benchmark [33] framework (which also provided the example in Section 2.1).…”

“…We investigate the execution time and scalability of the generated query plans for incremental query evaluation workloads using the open-source Train Benchmark project [33]. The results show that using basic optimization techniques avoiding Cartesian products already results in efficient query plans that scale for models with 9M+ elements, while applying further optimizations (e.g.…”

Model-Driven Engineering of an OpenCypher Engine: Using Graph Queries to Compile Graph Queries

“…This gives a rich set of features for users to model their specific domain in a natural way. Graph databases are able to store property graphs and query their contents by matching complex graph patterns, which would otherwise be cumbersome to define and/or inefficient to evaluate on traditional relational databases [39].…”