Time to clean your test objectives

Marcozzi, Michaël; Bardin, Sébastien; Kosmatov, Nikolaï; Papadakis, Mike; Prévosto, Virgile; Correnson, Loïc

doi:10.1145/3180155.3180191

Cited by 30 publications

(16 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Perhaps the closest work to ours, is from Marcozzi et al [38], which attempts to identify subsumed mutants using verification techniques (such as weakest precondition). While Marcozzi et al's approach is particularly powerful, it targets weak mutation and not strong as we do.…”

Section: Related Workmentioning

confidence: 99%

Cerebro: Static Subsuming Mutant Selection

Garg

Ojdanić

Degiovanni

et al. 2023

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Mutation testing research has indicated that a major part of its application cost is due to the large number of low utility mutants that it introduces. Although previous research has identified this issue, no previous study has proposed any effective solution to the problem. Thus, it remains unclear how to mutate and test a given piece of code in a best effort way, i.e., achieving a good trade-off between invested effort and test effectiveness. To achieve this, we propose Cerebro, a machine learning approach that statically selects subsuming mutants, i.e., the set of mutants that resides on the top of the subsumption hierarchy, based on the mutants' surrounding code context. We evaluate Cerebro using 48 and 10 programs written in C and Java, respectively, and demonstrate that it preserves the mutation testing benefits while limiting application cost, i.e., reduces all cost application factors such as equivalent mutants, mutant executions, and the mutants requiring analysis. We demonstrate that Cerebro has strong inter-project prediction ability, which is significantly higher than two baseline methods, i.e., supervised learning on features proposed by state-of-the-art, and random mutant selection. More importantly, our results show that Cerebro's selected mutants lead to strong tests that are respectively capable of killing 2 times higher than the number of subsuming mutants killed by the baselines when selecting the same number of mutants. At the same time, Cerebro reduces the cost-related factors, as it selects, on average, 68% fewer equivalent mutants, while requiring 90% fewer test executions than the baselines.

show abstract

Section: Related Workmentioning

confidence: 99%

Cerebro: Static Subsuming Mutant Selection

Garg

Ojdanić

Degiovanni

et al. 2023

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Regarding RQ3, while it is natural to expect benefits of a combination of different analyses, the results were somewhat surprising. Unlike in the previous work for other (non dataflow) criteria [3,11], the weakest precondition based technique brings only very slight benefits (see columns M NA,Eq,VA -M NA,Eq,VA,WP ). We believe it is due to the complex nature of dataflow criteria where infeasibility is less likely to be detected by local reasoning.…”

Section: Implementation and Evaluationmentioning

confidence: 86%

“…Recent research showed that value analysis and weakest precondition calculus can be efficient to detect polluting test objectives for several (non dataflow) criteria [3,11] expressed in HTOL. For dataflow criteria, model-checking was applied to detect infeasible test objectives [14].…”

Section: Background and Motivating Examplementioning

confidence: 99%

“…Yet it is crucial to avoid a waste of time during test generation (trying to cover polluting test objectives) and to allow a correct computation of coverage ratios (by ignoring polluting objectives in the total number of objectives). While applying dataflow criteria for testing [2,[6][7][8][9][10] and detection of polluting test objectives for simpler criteria (see [3,11] for some recent results and related work) were previously studied, evaluating and combining various program analysis techniques for their detection for dataflow criteria-the purpose of this work-was not investigated.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Detection of Polluting Test Objectives for Dataflow Criteria

Martin

Kosmatov

Prévosto

et al. 2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

“…There are many studies aiming at making the mutation score metric accurate either by using specific mutant types [9], or by detecting equivalent mutants [10], [11], i.e., mutants that cannot be killed by any test case because they are semantically equivalent to the original program, or by eliminating redundant mutants [12], [13], i.e., mutants that are killed "collaterally" whenever other mutants are killed (subsumed by the subsuming mutants). Yet, little research has focused on measuring the effectiveness of test suites with respect to particular program changes or commits.…”

Section: Introductionmentioning

confidence: 99%

Commit-Aware Mutation Testing

Laurent

Ojdanić

et al. 2020

2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)

Self Cite

View full text Add to dashboard Cite

In Continuous Integration, developers want to know how well they have tested their changes. Unfortunately, in these cases, the use of mutation testing is suboptimal since mutants affect the entire set of program behaviours and not the changed ones. Thus, the extent to which mutation testing can be used to test committed changes is questionable. To deal with this issue, we define commit-relevant mutants; a set of mutants that affect the changed program behaviours and represent the commit-relevant test requirements. We identify such mutants in a controlled way, and check their relationship with traditional mutation score (score based on the entire set of mutants or on the mutants located on the commits). We conduct experiments in both C and Java, using 83 commits, 2,253,610 mutants from 25 projects. Our findings reveal that there is a relatively weak correlation (Kendall/Pearson 0.15-0.4) between the sought (commit-relevant) and traditional mutation scores, indicating the need for a commit-aware test assessment metric. Our analysis also shows that traditional mutation is far from the envisioned case as it loses approximately 50%-60% of the commit-relevant mutants when analysing 5-25 mutants. More importantly, our results demonstrate that traditional mutation has approximately 30% lower chances of revealing commit-introducing faults than commit-aware mutation testing.

show abstract

Time to clean your test objectives

Cited by 30 publications

References 65 publications

Cerebro: Static Subsuming Mutant Selection

Cerebro: Static Subsuming Mutant Selection

Detection of Polluting Test Objectives for Dataflow Criteria

Commit-Aware Mutation Testing

Contact Info

Product

Resources

About