Mutation testing on an object-oriented framework: An experience report

Abstract

Context: The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners. Objective: This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective. Method: In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT. Results: Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field. Conclusion: We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.

Class mutation operators for C++ object-oriented systems

“…Thus, ten operators in the set did not create any mutant in this experiment. Therefore, the results obtained with traditional operators for other languages should not be compared with the results presented here, but with studies around class mutants, where similar numbers can be found [11,16]. In this sense, the practical application of the technique is more affordable.…”

“…Other common conclusion in related studies is the high equivalence percentage: while traditional operators generate between 5-15% of equivalence, this percentage was largely increased using class operators in the same programs [14,16]. However, the statistics calculated are rather different depending on the operators and applications analysed: from 45,4% [16] to 86.45% [11] in recent works.…”

“…Regarding tinyxml2, all the mutants generated were considered; in contrast for xmlrpc++, the mutants related with log and error reports were discarded for being secondary functionalities as well as the classes uncovered by the test suite were excluded (the terms discarded and uncovered are taken from [16]). The classification of the mutants obtained is depicted in Table 3 and 4 for tinyxml2 and xmlrpc++ respectively.…”

“…However, the statistics calculated are rather different depending on the operators and applications analysed: from 45,4% [16] to 86.45% [11] in recent works. This study reports an average percentage of 36,9%, which is lower than the aforementioned results, but corroborates that equivalence is a prominent issue.…”

“…Firstly, it should be noted that the amount of mutants stemming from class mutation operators is much lower than for traditional statement-level operators, as stated in [14] and also in [16], where the number of class-level mutants produced was 60% lower than their traditional counterparts; the language constructs modified by these operators are not as much used as arithmetic or logical operators. Thus, ten operators in the set did not create any mutant in this experiment.…”