Reliable testing: detecting state-polluting tests to prevent test dependency

Gyori, Alex; Shi, August; Hariri, Farah; Marinov, Darko

doi:10.1145/2771783.2771793

Cited by 72 publications

(26 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Developers can obtain test dependencies for a test suite by (1) manually specifying the test dependencies, or (2) using automatic tools to compute the test dependencies [26,29,44,65,72]. For our evaluation, we obtain test dependencies through the latter approach, using automatic tools, because we want to evaluate the scenario of how any developer can benefit from our enhanced algorithms without having to manually specify test dependencies.…”

Section: Computing Test Dependenciesmentioning

confidence: 99%

“…Various techniques have been proposed to detect test dependencies automatically. PolDet [29] finds tests that pollute the shared state -tests that modify some location on the heap shared across tests or on the file system. However, the tests that PolDet detects do not fail themselves, and while they pollute shared state, no other tests currently in the test suite may fail due to that polluted shared state.…”

Section: Techniques To Detect Test Dependenciesmentioning

confidence: 99%

See 1 more Smart Citation

Dependent-test-aware regression testing techniques

Lam

Shi

Oei

et al. 2020

Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures. To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of OD tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with OD tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 82% of human-written test suites and 100% of automatically-generated test suites with OD tests have at least one flaky-test failure. We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and apply our approach to 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce orders with different permutations or extra tests. Our evaluation shows that, in comparison to the orders produced by unenhanced algorithms, the orders produced by enhanced algorithms (1) have overall 80% fewer flaky-test failures due to OD tests, and (2) may add extra tests but run only 1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky-test failures with only a minor slowdown to run the tests.

show abstract

Section: Computing Test Dependenciesmentioning

confidence: 99%

Section: Techniques To Detect Test Dependenciesmentioning

confidence: 99%

Dependent-test-aware regression testing techniques

Lam

Shi

Oei

et al. 2020

Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

show abstract

“…Bell et al [11] provide a dependency detection tool (Electrictest), and compare it to other state of the art tools, getting a similar fault detection rate with lower slowdown. Gyori et al [12] introduce formally the concept of test Pollution problem and present a technique (called POLDET) that detects in execution time the "polluting" tests. Another technique that addresses the test dependence problem is in [13], where Gambi et al provide an evolution of their prior approach (ElectricTest), and test it empirically with good results: they discover the known dependences and find another one that previous tests and tools did not discover.…”

Section: -29238-6_22mentioning

confidence: 99%

RETORCH: Resource-Aware End-to-End Test Orchestration

Augusto¹,

Morán²,

Bertolino³

et al. 2019

Communications in Computer and Information Science

View full text Add to dashboard Cite

Continuous integration practices introduce incremental changes in the code to both improve the quality and add new functionality. These changes can introduce faults that can be timely detected through continuous testing by automating the test cases and re-executing them at each code change. However, reexecuting all test cases at each change may not be always feasible, especially for those test cases that make heavy use of resources thoroughly like End-to-End test cases that need a complex test infrastructure. This paper is focused on optimizing the usage of the resources employed during End-to-End testing (e.g., storage, memory, web servers or tables of a database, among others) through a resourceaware test orchestration technique in the context of continuous integration in the cloud. In order to optimize both the cost/usage of resources and the execution time, the approach proposes to i) identify the resources required by the End-to-End test cases, ii) group together those tests that need the same resources, iii) deploy the tests in both dependency isolated and elastic environments, and iv) schedule their parallel execution in several machines.

show abstract

“…Our test state representation is closely related to the heap representation used by Gyori et al [18]. They store the portions of the concrete heap accessible from static fields of test classes.…”

Section: Related Workmentioning

confidence: 99%

Fine-Grained Test Minimization

Vahabzadeh¹,

Stocco²,

Mesbah³

2018

EasyChair Preprints

View full text Add to dashboard Cite

As a software system evolves, its test suite can accumulate redundancies over time. Test minimization aims at removing redundant test cases. However, current techniques remove whole test cases from the test suite using test adequacy criteria, such as code coverage. This has two limitations, namely (1) by removing a whole test case the corresponding test assertions are also lost, which can inhibit test suite effectiveness, (2) CCS CONCEPTS• Software and its engineering → Software testing and debugging;

show abstract

Reliable testing: detecting state-polluting tests to prevent test dependency

Cited by 72 publications

References 16 publications

Dependent-test-aware regression testing techniques

Dependent-test-aware regression testing techniques

RETORCH: Resource-Aware End-to-End Test Orchestration

Fine-Grained Test Minimization

Contact Info

Product

Resources

About