Root causing flaky tests in a large-scale industrial setting

Lam, Wing; Godefroid, Patrice; Nath, Suman; Santhiar, Anirudh; Thummalapenta, Suresh

doi:10.1145/3293882.3330570

Cited by 102 publications

(66 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, several research works have checked test procedures in a broad variety of platforms from Android cell phones 65 to large‐scale industrial settings 9 in order to identify the root causes of flaky tests.…”

Section: Results and Analysismentioning

confidence: 99%

“…Another research work published in 2019 reports 5% flaky tests in Randoop (Obtained from five open‐source Java projects with a total of 78 versions) 8 . Moreover, Microsoft confirmed flaky tests to be the second most important reason (out of 10 reasons) adversely affecting software deployment time 9 . Moreover, it was reported in Reference 10 that 18% of test suite executions lead to failure and that 13% of these failures are caused by flaky tests.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software‐defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.

show abstract

Section: Results and Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Existing techniques for detecting flaky tests rely on rerunning tests: if we can witness two different outcomes (passing and failing) from the same test on the same version of the codebase, then surely that test is flaky. Even state-of-theart techniques like NonDex [16] and iDFlakies [17] that inject additional non-determinism still rely on re-running each test case many times. However, rerunning tests can be quite expensive.…”

Section: Introductionmentioning

confidence: 99%

FlakeFlagger: Predicting Flakiness Without Rerunning Tests

Alshammari¹,

Morris²,

Hilton³

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

When developers make changes to their code, they typically run regression tests to detect if their recent changes (re)introduce any bugs. However, many tests are flaky, and their outcomes can change non-deterministically, failing without apparent cause. Flaky tests are a significant nuisance in the development process, since they make it more difficult for developers to trust the outcome of their tests, and hence, it is important to know which tests are flaky. The traditional approach to identify flaky tests is to rerun them multiple times: if a test is observed both passing and failing on the same code, it is definitely flaky. We conducted a very large empirical study looking for flaky tests by rerunning the test suites of 24 projects 10,000 times each, and found that even with this many reruns, some previously identified flaky tests were still not detected. We propose FlakeFlagger, a novel approach that collects a set of features describing the behavior of each test, and then predicts tests that are likely to be flaky based on similar behavioral features. We found that FlakeFlagger correctly labeled as flaky at least as many tests as a state-of-the-art flaky test classifier, but that FlakeFlagger reported far fewer false positives. This lower false positive rate translates directly to saved time for researchers and developers who use the classification result to guide more expensive flaky test detection processes. Evaluated on our dataset of 23 projects with flaky tests, FlakeFlagger outperformed the prior approach (by F1 score) on 16 projects and tied on 4 projects. Our results indicate that this approach can be effective for identifying likely flaky tests prior to running time-consuming flaky test detectors.

show abstract

“…these tests can mislead developers to debug their recent changes while the failures can be due to a variety of reasons unrelated to the changes. Many software organizations have reported flaky tests as one of their biggest problems in software development, including Apple [18], Facebook [5,10], Google [8,30,31,43,48], Huawei [16], Microsoft [11,12,20,21], and Mozilla [40].…”

Section: Introductionmentioning

confidence: 99%

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Wei

Xie

et al. 2021

Tools and Algorithms for the Construction and Analysis of Systems

Self Cite

View full text Add to dashboard Cite

Software developers frequently check their code changes by running a set of tests against their code. Tests that can nondeterministically pass or fail when run on the same code version are called flaky tests. These tests are a major problem because they can mislead developers to debug their recent code changes when the failures are unrelated to these changes. One prominent category of flaky tests is order-dependent (OD) tests, which can deterministically pass or fail depending on the order in which the set of tests are run. By detecting OD tests in advance, developers can fix these tests before they change their code. Due to the high cost required to explore all possible orders (n! permutations for n tests), prior work has developed tools that randomize orders to detect OD tests. Experiments have shown that randomization can detect many OD tests, and that most OD tests depend on just one other test to fail. However, there was no analysis of the probability that randomized orders detect OD tests. In this paper, we present the first such analysis and also present a simple change for sampling random test orders to increase the probability. We finally present a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test, while running substantially fewer orders and tests than simply running all test pairs.

show abstract

Root causing flaky tests in a large-scale industrial setting

Cited by 102 publications

References 34 publications

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

FlakeFlagger: Predicting Flakiness Without Rerunning Tests

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Contact Info

Product

Resources

About