Assessment of class mutation operators for C++ with the MuCPP mutation system

Abstract

Context: Mutation testing has been mainly analyzed regarding traditional mutation operators involving structured programming constructs common in mainstream languages, but mutations at the class level have not been assessed to the same extent. This fact is noteworthy in the case of C++, despite being one of the most relevant languages including object-oriented features. Objective: This paper provides a complete evaluation of class operators for the C++ programming language. MuCPP, a new system devoted to the application of mutation testing to this language, was developed to this end. This mutation system implements class mutation operators in a robust way, dealing with the inherent complexity of the language. Method: MuCPP generates the mutants by traversing the abstract syntax tree of each translation unit with the Clang API, and stores mutants as branches in the Git version control system. The tool is able to detect duplicate mutants, avoid system headers, and drive the compilation process. Then, MuCPP is used to conduct experiments with several open-source C++ programs. Results: The improvement rules listed in this paper to reduce unproductive class mutants have a significant impact in the computational cost of the technique. We also calculate the quantity and distribution of mutants generated with class operators, which generate far fewer mutants than their traditional counterparts. Conclusions: We show that the tests accompanying these programs cannot detect faults related to particular object-oriented features of C++. In order to increase the mutation score, we create new test scenarios to kill the surviving class mutants for all the applications. The results confirm that, while traditional mutation operators are still needed, class operators can complement them and help testers further improve the test suite.

Mutation Testing Advances: An Analysis and Survey

“…Delgado-Pérez [7,82] conducted an evaluation of class-level mutant operators for C++. Based on the results, they propose a C++ mutation tool, MuCPP, which generates mutants by traversing the abstract syntax tree of each translation unit with the Clang API.…”

“…Its primary application level is unit testing but several advances have been made in order to support other levels, i.e., specification [2], design [3], integration [4] and system levels [5]. The method has been applied on the most popular programming languages such as C [6], C++ [7], C# [8], Java [9], JavaScript [10], Ruby [11] including specification [2] and modelling languages [12]. It has also been adapted for the most popular programming paradigms such as Object-Oriented [13], Functional [14], aspect-oriented and declarative-oriented [15,16] programming.…”

Assessment of C++ object‐oriented mutation operators: A selective mutation approach

“…Each block includes those operators addressing similar syntactic elements [4,6]. These mutation operators have been implemented in the mutation system MuCPP [25]. We should note that MuCPP has been optimised towards increasing the percentage of valid mutants (complying with the grammar rules of the language) over the total number of mutants.…”

“…In previous work using class operators for C++ [6], the mutation score was calculated around 2 case studies and the test suites distributed with them, showing which operators spawned more mutants that remained undetected. Several improvement rules for these operators where proposed later, analysing their impact in the resources required to generate and execute the mutants for 5 applications [25].…”

Coverage-based quality metric of mutation operators for test suite improvement

“…The set of class operators for C++ encompassed both adapted operators for Java (Ma et al, 2002)/C# (Derezińska, 2006) and tailored operators to C++ features. A revised list of these operators has been implemented in the mutation tool called MuCPP 2 , where they underwent a thorough study to increase the ratio of useful mutants generated (Delgado-Pérez et al, 2017a). This set of C++ class operators, which was previously assessed following a selective mutation approach (Delgado-Pérez et al, 2017b), will be analysed in the conducted experiments in the next section.…”