iFixFlakies: a framework for automatically fixing order-dependent flaky tests

Shi, August; Lam, Wing; Oei, Reed; Xie, Tao; Marinov, Darko

doi:10.1145/3338906.3338925

Cited by 84 publications

(46 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that test orders may contain more tests besides polluters and cleaners for a victim v, but these other tests do not modify the relevant state and do not affect whether v passes or not in any order. Precise definitions for these tests are in our previous work [37]. Figure 1 shows a snippet of a victim test, testMRAppMasterSuccessLock (in short testV), from the widely used Hadoop project [1].…”

Section: Background and Examplementioning

confidence: 99%

“…Specifically, we first determine whether an OD test t is a victim or a brittle by running the test in isolation, i.e., just t , by itself 10 times: if t always passes, it is considered a victim (although it may be an ND test); if t always fails, it is considered a brittle (although it may be an ND test); and if t sometimes passes and sometimes fails, it is definitely an ND, not OD, test. This approach was proposed for iFixFlakies [37], and using 10 runs is a common industry practice to check whether a test is flaky [31,40].…”

Section: Determining Test Outcome Without Running a Test Ordermentioning

confidence: 99%

“…Our iFixFlakies work [37] has studied the causes of failures for OD tests. We find that the vast majority of OD tests are related to pairs of tests, i.e., each OD test would pass or fail due to the sharing of some global state with just one other test.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Wei

Xie

et al. 2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Software developers frequently check their code changes by running a set of tests against their code. Tests that can nondeterministically pass or fail when run on the same code version are called flaky tests. These tests are a major problem because they can mislead developers to debug their recent code changes when the failures are unrelated to these changes. One prominent category of flaky tests is order-dependent (OD) tests, which can deterministically pass or fail depending on the order in which the set of tests are run. By detecting OD tests in advance, developers can fix these tests before they change their code. Due to the high cost required to explore all possible orders (n! permutations for n tests), prior work has developed tools that randomize orders to detect OD tests. Experiments have shown that randomization can detect many OD tests, and that most OD tests depend on just one other test to fail. However, there was no analysis of the probability that randomized orders detect OD tests. In this paper, we present the first such analysis and also present a simple change for sampling random test orders to increase the probability. We finally present a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test, while running substantially fewer orders and tests than simply running all test pairs.

show abstract

Section: Background and Examplementioning

confidence: 99%

Section: Determining Test Outcome Without Running a Test Ordermentioning

confidence: 99%

See 1 more Smart Citation

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Wei

Xie

et al. 2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…The efficiency goal is a trade-off against the correctness goal. [61] refers to test as a state-setter and test in a positive dependency as a brittle. It also refers to test as a polluter and in a negative dependency as a victim.…”

Section: Dependent-test-aware Regression Testing Techniquesmentioning

confidence: 99%

“…Our work aims at different regression testing techniques and can be used in conjunction with Nemo. iFixFlakies [61] is a tool for automatically fixing OD tests. iFixFlakies relies on finding code from "helpers", which are tests in the existing test suite that prevent OD-test failures when run before an OD test, to fix the OD tests.…”

Section: Techniques For Managing Flaky Testsmentioning

confidence: 99%

Dependent-test-aware regression testing techniques

Lam

Shi

Oei

et al. 2020

Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures. To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of OD tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with OD tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 82% of human-written test suites and 100% of automatically-generated test suites with OD tests have at least one flaky-test failure. We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and apply our approach to 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce orders with different permutations or extra tests. Our evaluation shows that, in comparison to the orders produced by unenhanced algorithms, the orders produced by enhanced algorithms (1) have overall 80% fewer flaky-test failures due to OD tests, and (2) may add extra tests but run only 1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky-test failures with only a minor slowdown to run the tests.

show abstract

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software‐defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.

show abstract

iFixFlakies: a framework for automatically fixing order-dependent flaky tests

Cited by 84 publications

References 42 publications

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Dependent-test-aware regression testing techniques

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

Contact Info

Product

Resources

About