Revisiting the relationship between fault detection, test adequacy criteria, and test set size

Chen, Yiqun T.; Gopinath, Rahul; Tadakamalla, Anita; Ernst, M.; Holmes, Reid; Ammann, Paul; Just, René

doi:10.1145/3324884.3416667

Cited by 39 publications

(24 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To investigate the possible changes in grades for our first research question, we designed our test suite sampler to simulate the iterative development of various individual test suites. In order to do this, we utilised the suite growth technique described by Chen et al [14], in which a test suite is extended by randomly selecting an additional test that increases a given criterion, generating a new suite whenever a test is added. The first test suite is created by simply randomly selecting any test from the whole set.…”

Section: Grading Test Suitesmentioning

confidence: 99%

“…As with the mutation score property, we only use mutants with unique combinations of passing tests, to avoid bias from different proportions of similar mutants. Since it is possible for suites to detect every mutant, we use a "stacking" approach to continue growing the suite [14,15]; the suite's current mutation score is reset to zero by the generator, so every available test that detects any mutants can be selected. Once there is only one unselected test, our generator ends the generation run.…”

Section: Grading Test Suitesmentioning

confidence: 99%

See 1 more Smart Citation

Diagnosability, Adequacy & Size: How Test Suites Impact Autograding

Clegg¹,

McMinn²

2022

Proceedings of the Annual Hawaii International Conference on System Sciences

Self Cite

View full text Add to dashboard Cite

Automated grading is now prevalent in software engineering courses, typically assessing the correctness of students' programs using automated test suites. However, deficiencies in test suites could result in inconsistent grading. As such, we investigate how different test suites impact grades, and the extent to which their observable properties influence these grades. We build upon existing work, using students' solution programs, and test suites that we constructed using a sampling approach. We find that there is a high variation in grades from different test suites, with a standard deviation of ∼10.1%. We further investigate how several properties of test suites influence these grades, including the number of tests, coverage, ability to detect other faults, and uniqueness. We use our findings to provide tutors with strategies for building test suites that evaluate students' software with consistency. These strategies include constructing test suites with high coverage, writing unique and diverse tests which evaluate solutions' correctness in different ways, and to run the tests against artificial faults to determine their quality.

show abstract

Section: Grading Test Suitesmentioning

confidence: 99%

Section: Grading Test Suitesmentioning

confidence: 99%

Diagnosability, Adequacy & Size: How Test Suites Impact Autograding

Clegg¹,

McMinn²

2022

Proceedings of the Annual Hawaii International Conference on System Sciences

Self Cite

View full text Add to dashboard Cite

show abstract

“…An alternative approach that addresses this limitation is mutation analysis, which systematically seeds artificial faults, called mutants, into a program and measures a test suite's ability to detect them [4]. Mutation analysis addresses is widely considered the best approach for evaluating test suite efficacy [5], [6], [7]. Mutation testing is an iterative testing approach that builds on top of mutation analysis and uses undetected mutants as concrete test goals to guide the testing process.…”

Section: Introductionmentioning

confidence: 99%

Practical Mutation Testing at Scale: A view from Google

Petrović

Ivanković

Just

2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem-both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 150,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment; even existing solutions to speed up mutation analysis are insufficient to make it computationally feasible at such a scale. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas:(1) mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base;(2) mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. This paper empirically validates the proposed approach by analyzing its effectiveness in a code-review-based setting, used by more than 24,000 developers on more than 1,000 projects. The results show that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable to industrial settings of any size.

show abstract

“…However, we know that coverage can often be quickly saturated [14], and that adequate coverage is a necessary, but not sufficient condition for all bugs to be detected. Indeed, fuzzing researchers are aware of the limitations of coverage as a proxy measure [15].…”

Section: Introductionmentioning

confidence: 99%

“…Mutation analysis is free of all the above problems that we identified. For example, mutation analysis is much harder to saturate than code coverage [14], and is more robust than various forms of coverage as a proxy for the fault revealing power of the test suite.…”

Section: Introductionmentioning

confidence: 99%