A framework for evaluating tool support for co-evolution of modeling languages, tools and models

Abstract

We present a framework for evaluating language workbenches’ capabilities for co-evolution of graphical modeling languages, modeling tools and models. As with programming, maintenance tasks such as language refinement and enhancement typically account for more work than the initial development phase. Modeling languages have the added challenge of keeping tools and existing models in step with the evolving language. As domain-specific modeling languages and tools have started to be used widely, thanks to reports of significant productivity improvements, some language workbench users have indeed reported problems with co-evolution of tools and models. Our tool-agnostic evaluation framework aims to cover changes across the whole language definition: the abstract syntax, concrete syntax, and constraints. Change impact is assessed for knock-on effects within the language definition, the modeling tools, semantics via generators, and existing models. We demonstrate the viability of the framework by evaluating MetaEdit+, EMF/Sirius and Jjodel, providing a detailed evaluation process for others to repeat with their tools. The results of the evaluation show differences among the tools: from editors not opening correctly or at all, through highlighting of items requiring manual intervention, to fully automatic updates of languages, models and editors. We call for industry to evaluate their tool choices with the framework, tool developers to extend their tool support for co-evolution, and researchers to refine the evaluation framework and evaluations presented.