Model-based mutation testing—Approach and case studies

Abstract

This paper rigorously introduces the concept of model-based mutation testing (MBMT) and positions it in the landscape of mutation testing. Two elementary mutation operators, insertion and omission, are exemplarily applied to a hierarchy of graph-based models of increasing expressive power including directed graphs, event sequence graphs, finite-state machines and statecharts. Test cases generated based on the mutated models (mutants) are used to determine not only whether each mutant can be killed but also whether there are any faults in the corresponding system under consideration (SUC) developed based on the original model. Novelties of our approach are: (1) evaluation of the fault detection capability (in terms of revealing faults in the SUC) of test sets generated based on the mutated models, and (2) superseding of the great variety of existing mutation operators by iterations and combinations of the two proposed elementary operators. Three case studies were conducted on industrial and commercial real-life systems to demonstrate the feasibility of using the proposed MBMT approach in detecting faults in SUC, and to analyze its characteristic features. Our experimental data suggest that test sets generated based on the mutated models created by insertion operators are more effective in revealing faults in SUC than those generated by omission operators. Worth noting is that test sets following the MBMT approach were able to detect faults in the systems that were tested by manufacturers and independent testing organizations before they were released.

Mutation Testing Advances: An Analysis and Survey

“…Here, we briefly discuss these approaches. For a detailed description and discussion on this subject, we point the reader to the specialised survey of Belli et al [30].…”

“…Despite this, we see a growing interest towards this direction. There is a recent dedicated survey on this subject [30] and multiple high profile publications over the last couple of years. Additionally, very recently efficient and scalable tools have been built, e.g., the VIBeS tool [12], which hopefully will push the research in this area further.…”

“…These are: a survey on the Equivalent mutant problem, by Madeyski et al. [28] (in 2014), a systematic mapping of mutation-based test generation by Souza et al [29] (in 2014), a survey on model-based mutation testing [30] (in 2016) and a systematic review on search-based mutation testing by Silva et al [31] (in 2017). However, none of these covers the whole spectrum of advances from 2009.…”

Automata Language Equivalence vs. Simulations for Model-Based Mutant Equivalence: An Empirical Evaluation

“…Regarding behavioural models, Aichernig et al [8] developed a mutation-based test generation technique for state machines. Belli et al [26] compare mutation-testing strategies when applied on event-based and state-based models, and found that both had similar effectiveness.…”

Mutation Testing Approach to Negative Testing

“…Nowadays, the application area of mutation testing is not limited to assessing test cases at implementation level only. A number of works have shown examples of applying it to different formalisms, at various levels of software development, and for assessment, as well as for generation of test cases [14][15][16][17][18][19][20][21][22][23][24]. Works dedicated to mutation testing-based generation of test cases usually share the idea of mutating a system, its model, or specification and then selecting test cases being able to detect the mutants [15,18,19,[23][24][25][26].…”