STAMINA: a competition to encourage the development and assessment of software model inference techniques

Abstract

Models play a crucial role in the development and maintenance of software systems, but are often neglected during the development process due to the considerable manual effort required to produce them. In response to this problem, numerous techniques have been developed that seek to automate the model generation task with the aid of increasingly accurate algorithms from the domain of Machine Learning. From an empirical perspective, these are extremely challenging to compare; there are many factors that are difficult to control (e.g. the richness of the input and the complexity of subject systems), and numerous practical issues that are just as troublesome (e.g. tool availability). This paper describes the StaMinA (State Machine Inference Approaches) competiton, that was designed to address these This work is supported by the EPSRC STAMINA project (EP/H002456/1), the EPSRC REGI project (EP/F065825/1), the Regional Government of Wallonia (GISELE project, RW Conv. 616425) (2013) 18:791-824 problems. The competition attracted numerous submissions, many of which were improved or adapted versions of techniques that had not been subjected to extensive empirical evaluations, and had not been evaluated with respect to their ability to infer models of software systems. This paper shows how many of these techniques substantially improve on the state of the art, providing insights into some of the factors that could underpin the success of the best techniques. In a more general sense it demonstrates the potential for competitions to act as a useful basis for empirical software engineering by (a) spurring the development of new techniques and (b) facilitating their comparative evaluation to an extent that would usually be prohibitively challenging without the active participation of the developers.

PAutomaC: a probabilistic automata and hidden Markov models learning competition

“…First of all, the root decision shows that their method performs best on target distributions with a small alphabet (< 18). An interesting question is whether this can be linked to the known problems of state merging methods for non-probabilistic automata on large alphabets [77]. Secondly, from this tree it is very clear that the type of generating distribution has a large effect on the performance.…”

Inferring Extended Finite State Machine models from software executions

“…-FSM inference: Several techniques exist to infer FSMs from traces [2,12,32,50,6,31,42]. Although these can be quite accurate, they fail to capture the crucial relationship between the events and the data-state.…”

“…Numerous inference techniques were developed, spurred by several competitions that encouraged novel techniques to infer models from training sets made available over the internet [31,50]. Evidence-Driven State Merging (EDSM) is one of the most popular and accurate inference techniques to emerge.…”

“…Evidence-Driven State Merging (EDSM) is one of the most popular and accurate inference techniques to emerge. It won the Abbadingo competition [31] and was used as a baseline for the recent STAMINA competition [50]. We use the basic steps of the EDSM algorithm as a basis for our EFSM inference algorithm.…”

“…Most techniques are dynamic; they take as input a set of execution traces of the system in question and infer models that generalise upon them. These have largely focussed either on the inference of sequential models in the form of Finite State Machines (FSMs) [2,12,32,50,6,31], or data pre-and post-conditions, as espoused by tools such as Daikon [17].…”