Sufficient mutation operators for measuring test effectiveness

Namin, Akbar Siami; Andrews, James H.; Murdoch, Duncan J.

doi:10.1145/1368088.1368136

Cited by 118 publications

(101 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Responding to this challenge, Offutt et al studied the effectiveness of a subset of mutation operators, revealing that this smaller group of sufficient mutation operators could be applied without a substantial loss of information [14]. While the actual subset that can be employed depends on the programming language, this paper considers the following set of mutation operators which are supported by an established mutation testing tool [6], [7] and commonly used in previous experiments [15], [16] …”

Section: Background On Mutation Analysismentioning

confidence: 99%

“…Previous studies on the reduction of mutation operators (e.g., [14], [16]) did not take their definition into account and considered the operators to be atomic. This means that, for instance, a replacement operator was either applied with all valid replacements or it was excluded.…”

Section: A Detailed View Of the Cor Operatormentioning

confidence: 99%

“…Previous studies revealed that a subset of all applicable mutation operators is sufficient to achieve a meaningful result [14], [16]. Regarding the operators to be atomic, these studies focused on reducing the number of mutation operators without incurring a major loss in the accuracy of the mutation score.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

Just

Kapfhammer

Schweiggert

2012

2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

View full text Add to dashboard Cite

Abstract-Mutation analysis is an unbiased and powerful method for assessing input values and test oracles. However, in comparison to other techniques, such as those that rely on code coverage, it is a computationally-expensive and time-consuming method, especially for large software systems. This high cost is due, in part, to the fact that many mutation operators generate redundant mutants that may both misrepresent the mutation score and increase the runtime of the mutation analysis process. After showing how the conditional operator replacement (COR) mutation operator can be defined in a redundant-free manner, this paper uses four real-world programs, ranging in size from 3,000 to nearly 40,000 lines of code, to show the prevalence of redundant mutants. Focusing on the conditional operator replacement (COR) and relational operator replacement (ROR) mutation operators that create 41% of all mutants in the chosen programs, the case study reveals that the removal of redundant mutants reduces the runtime of mutation analysis by up to 34%. Additional empirical results show that redundant mutants can lead to a mutation score that is misleadingly overestimated by as much as 10%. Overall, this paper convincingly demonstrates that it is possible to improve the effectiveness and efficiency of a mutation analysis system by identifying and removing redundant mutants.

show abstract

Section: Background On Mutation Analysismentioning

confidence: 99%

Section: A Detailed View Of the Cor Operatormentioning

confidence: 99%

See 1 more Smart Citation

Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

Just

Kapfhammer

Schweiggert

2012

2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

View full text Add to dashboard Cite

show abstract

“…There are many studies on fault detection eectiveness of model-based test generation using mutation analysis [1,2,19,22,23]. In order to further assess our approach we extended our SPLTestBench by a mutation framework and dened mutation operators for feature models, feature mappings, and the test model.…”

Section: Resultsmentioning

confidence: 99%

Automated Test Design for Boundaries of Product Line Variants

Weiβleder

Wartenberg

Lackner

2015

Testing Software and Systems

View full text Add to dashboard Cite

Abstract. Developing product lines is usually more ecient than developing single products because of the reuse of single components. Testing, however, has to consider complete, integrated systems. To prevent testing every product on system level, the whole product line should be analyzed with the aim of selecting distinguishing product behavior and a minimum of system products to test. In this paper, we present a model-based test design approach for testing the selected behavior of products, but also their deselected behavior. A major challenge of this approach is that the deselected behavior of a product is often not part of its behavioral model. Thus, we use the variability model to transform the behavioral model so that showing the exclusion of the deselected behavior is also covered by tests. We present the approach, a corresponding prototypical implementation, and our experiences using a set of examples.

show abstract

“…Two approaches to reduce the overall number of mutants, which incur a loss of information, are sampling subsets of mutants or restricting the set of operators that creates the mutants [30]. A lossless approach is to refine mutation operators to avoid those that result in redundant (i.e., semantically identical) mutants [20].…”

Section: Introductionmentioning

confidence: 99%

Efficient mutation analysis by propagating and partitioning infected execution states

Just

Ernst

2014

Proceedings of the 2014 International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Mutation analysis evaluates a testing technique by measuring how well it detects seeded faults (mutants). Mutation analysis is hampered by inherent scalability problems -a test suite is executed for each of a large number of mutants. Despite numerous optimizations presented in the literature, this scalability issue remains, and this is one of the reasons why mutation analysis is hardly used in practice.Whereas most previous optimizations attempted to statically reduce the number of executions or their computational overhead, this paper exploits information available only at run time to further reduce the number of executions.First, state infection conditions can reveal -with a single test execution of the unmutated program -which mutants would lead to a different state, thus avoiding unnecessary test executions. Second, determining whether an infected execution state propagates can further reduce the number of executions. Mutants that are embedded in compound expressions may infect the state locally without affecting the outcome of the compound expression. Third, those mutants that do infect the state can be partitioned based on the resulting infected state -if two mutants lead to the same infected state, only one needs to be executed as the result of the other can be inferred.We have implemented these optimizations in the Major mutation framework and empirically evaluated them on 14 open source programs. The optimizations reduced the mutation analysis time by 40% on average.

show abstract

Sufficient mutation operators for measuring test effectiveness

Cited by 118 publications

References 26 publications

Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis?

Automated Test Design for Boundaries of Product Line Variants

Efficient mutation analysis by propagating and partitioning infected execution states

Contact Info

Product

Resources

About