The care and feeding of wild-caught mutants

Abstract

Mutation testing of a test suite and a program provides a way to measure the quality of the test suite. In essence, mutation testing is a form of sensitivity testing: by running mutated versions of the program against the test suite, mutation testing measures the suite's sensitivity for detecting bugs that a programmer might introduce into the program. This paper introduces a technique to improve mutation testing that we call wild-caught mutants; it provides a method for creating potential faults that are more closely coupled with changes made by actual programmers. This technique allows the mutation tester to have more certainty that the test suite is sensitive to the kind of changes that have been observed to have been made by programmers in real-world cases. CCS CONCEPTS • Software and its engineering → Software configuration management and version control systems; Software testing and debugging; Parsers;

Learning How to Mutate Source Code from Bug-Fixes

“…Stemming from the previous considerations by Brown et al [22], as well as from recent work aimed at learning bug repairs from an existing set of previous fixes [23], [24] and, more generally, from the successful applications of machine learning on code to support several SE tasks tasks [25]- [34], we conjecture that mutants can be automatically learned from previous fixes. We propose an approach for automatically learning mutants from actual bug fixes.…”

“…This is a very conservative estimation that does not consider the mutated predictions. Brown et al [22] achieved a compilability rate of 14%. Moreover, "the majority of failed compilations (64%) arise from simple parsing errors" [22], whereas we achieve a better-estimated compilability and a high percentage of syntactically correct predictions.…”

“…Brown's et al approach does not contemplate the synthesis of new identifiers (see Sec. 2.3 [22]); • Rather than extracting a single mutation pattern, we can learn co-occurrences and combinations of multiple mutations; • While the approach by Brown et al randomly applies mutation operators to any code location unless the user specifies a rule for that, our approach automatically applies, for a given piece of code, the mutation(s) that according to the learning might reflect likely bugs occurring in such a location. While limiting mutants only to the most suitable ones for each location might not be always necessary, because one can apply as many mutants as possible to increase fault detection, this could lead to an overestimate of a test suite effectiveness or to more effort to unnecessarily augment a test suite.…”

“…Taxonomies of mutants have been devised by taking typical bugs into account [15], [16], and even coping with specific domains [17]- [21]. Brown et al [22] leveraged bug-fixes to infer 7.5k types of mutation operators from diff patches. However, devising specific mutant taxonomies requires a substantial manual effort, and may not generalize across projects.…”

Mutation Testing Advances: An Analysis and Survey

“…Brown et al [102] proposed a technique to mine mutation operators from source code repositories. The intuition of this work is that by making mutants sintactically similar to real faults one can get semantically similar mutants.…”