Exploiting Model Morphology for Event-Based Testing

Belli, Fevzi; Beyazit, Mutlu

doi:10.1109/tse.2014.2360690

Cited by 9 publications

(17 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, we created a class "others" in which we list 9 papers that we found hard to classify in any of the other 4 test phases. These works can be divided into 3 groups: grammar-based testing [133,134,135], spreadsheet-related testing [89,136,137] and SQL-related testing [138,139,140].…”

Section: No Evaluationmentioning

confidence: 99%

A systematic literature review of how mutation testing supports quality assurance processes

Zhu

Panichella

Zaidman

2018

Software Testing Verif & Rel

View full text Add to dashboard Cite

Mutation testing has been very actively investigated by researchers since the 1970s, and remarkable advances have been achieved in its concepts, theory, technology, and empirical evidence.While the most influential realisations have been summarised by existing literature reviews, we lack insight into how mutation testing is actually applied. Our goal is to identify and classify the main applications of mutation testing and analyse the level of replicability of empirical studies related to mutation testing. To this aim, this paper provides a systematic literature review on the application perspective of mutation testing based on a collection of 191 papers published between 1981 and 2015. In particular, we analysed in which quality assurance processes mutation testing is used, which mutation tools and which mutation operators are employed. Additionally, we also investigated how the inherent core problems of mutation testing, ie, the equivalent mutant problem and the high computational cost, are addressed during the actual usage. The results show that most studies use mutation testing as an assessment tool targeting unit tests, and many of the supporting techniques for making mutation testing applicable in practice are still underdeveloped.Based on our observations, we made 9 recommendations for future work, including an important suggestion on how to report mutation testing in testing experiments in an appropriate manner. KEYWORDSapplication, mutation testing, systematic literature review 1 INTRODUCTION Mutation testing is defined by Jia and Harman [1] as a fault-based testing technique, which provides a testing criterion called the mutation adequacy score. This score can be used to measure the effectiveness of a test set in terms of its ability to detect faults [1]. The principle of mutation testing is to introduce syntactic changes into the original program to generate faulty versions (called mutants) according to well-defined rules (mutation operators) [2]. Mutation testing originated in the 1970s with works from Lipton [3], DeMillo et al. [4], and Hamlet [5] and has been a very active research field over the last few decades. The activeness of the field is in part evidenced by the extensive survey of more than 390 papers on mutation testing that Jia and Harman published in 2011 [1]. Jia and Harman's survey highlights the research achievements that have been made over the years, including the development of tools for a variety of languages and empirical studies performed [1]. Additionally, they highlight some of the actual and inherent problems of mutation testing, among others: (1) the high computational cost caused by generating and executing the numerous mutants and (2) the tremendous time-consuming human investigation required by the test oracle problem and equivalent mutant detection.While existing surveys (eg [1,2,6]) provide us with a great overview of the most influential realisations in research, we lack insight into how mutation testing is actually applied. Specifically, we are interested in analysing ...

show abstract

Section: No Evaluationmentioning

confidence: 99%

A systematic literature review of how mutation testing supports quality assurance processes

Zhu

Panichella

Zaidman

2018

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…The follow-on works of [44] by Belli and Guldali [68,69] extended the idea of using model checkers for test generation by generating a series of faulty models as inputs to a model checker to generate test cases, instead of solely one model. The approach represented by Offutt et al [70] thoroughly differs from the approach in the present paper for the following reasons explained in [71]. Offutt et al [70] used grammar-based models and mutation operators, namely for duplication, deletion, and replacement of nonterminal and terminals of a given grammar.…”

Section: Model-based Mutation Testingmentioning

confidence: 99%

“…Offutt et al [70] used grammar-based models and mutation operators, namely for duplication, deletion, and replacement of nonterminal and terminals of a given grammar. Belli and Beyazit [71] avoid the use of these operators for the following reasons:…”

Section: Model-based Mutation Testingmentioning

confidence: 99%

“…In [71], this drawback is exemplified using the regular grammar that is drawn from an example used by Ammann and Offutt [72]. Belli and Beyazit [71] have slightly, nevertheless equivalently, reformatted the example for saving space, where the nonterminal duplication mutant is a CFG.…”

Section: Model-based Mutation Testingmentioning

confidence: 99%

“…Belli and Beyazit [71] have slightly, nevertheless equivalently, reformatted the example for saving space, where the nonterminal duplication mutant is a CFG.…”

Section: Model-based Mutation Testingmentioning

confidence: 99%

See 2 more Smart Citations

Model-based mutation testing—Approach and case studies

Belli

Budnik

Hollmann³

et al. 2016

Science of Computer Programming

Self Cite

View full text Add to dashboard Cite

This paper rigorously introduces the concept of model-based mutation testing (MBMT) and positions it in the landscape of mutation testing. Two elementary mutation operators, insertion and omission, are exemplarily applied to a hierarchy of graph-based models of increasing expressive power including directed graphs, event sequence graphs, finite-state machines and statecharts. Test cases generated based on the mutated models (mutants) are used to determine not only whether each mutant can be killed but also whether there are any faults in the corresponding system under consideration (SUC) developed based on the original model. Novelties of our approach are: (1) evaluation of the fault detection capability (in terms of revealing faults in the SUC) of test sets generated based on the mutated models, and (2) superseding of the great variety of existing mutation operators by iterations and combinations of the two proposed elementary operators. Three case studies were conducted on industrial and commercial real-life systems to demonstrate the feasibility of using the proposed MBMT approach in detecting faults in SUC, and to analyze its characteristic features. Our experimental data suggest that test sets generated based on the mutated models created by insertion operators are more effective in revealing faults in SUC than those generated by omission operators. Worth noting is that test sets following the MBMT approach were able to detect faults in the systems that were tested by manufacturers and independent testing organizations before they were released.

show abstract