What is the Vocabulary of Flaky Tests?

Pinto, Gustavo; Miranda, Breno; Dissanayake, Supun; d’Amorim, Marcelo; Treude, Christoph; Bertolino, Antonia

doi:10.1145/3379597.3387482

Cited by 67 publications

(82 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, the literature of flaky tests has come with some useful conceptual and theoretical research projects such as establishing the life cycle of flaky tests 43 or standardizing the related vocabulary 44 . The review presented in this article takes the next step.…”

Section: Results and Analysismentioning

confidence: 99%

See 1 more Smart Citation

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software‐defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.

show abstract

Section: Results and Analysismentioning

confidence: 99%

“…Some universities have started checking their own products and test frameworks for flakiness 42 . Moreover, academic researchers have started investigating different aspects of flaky tests such as their life cycle 43 and the related vocabulary 44 . However, the academic research community is behind the industry in this area.…”

Section: Introductionmentioning

confidence: 99%

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Hence, rather than build a single model, we designed FlakeFlagger to construct a set of models based on seven different supervised learning algorithms: Decisions Tree (DT), Random Forest (RF), Support Vector Machine (SVM), Multilayer Perceptron (MLP), Naive Bayes (NB), Adaboosting (Ada), and K-Nearest Neighbor (KNN), using the Scikit-learn package [36]. This selection of models builds on flaky test classification prior work that considered DT, RF, SVM, KNN and NB models only, which found that RF performed best [12]. In our evaluation (described in the following section), we also found that the Random Forest had the best performance, and report results only for this model, but make all models available in our artifact [20], [21].…”

Section: B Flaky Tests Classifiermentioning

confidence: 99%

“…A promising alternative approach to detect flaky tests is to create a machine learning classifier that can distinguish between flaky and non-flaky tests [12], [19]. In such an approach, developers train a classifier using a known corpus of flaky and non-flaky tests, and then apply that classifier to a new codebase in order to detect new flaky tests.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

FlakeFlagger: Predicting Flakiness Without Rerunning Tests

Alshammari¹,

Morris²,

Hilton³

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

When developers make changes to their code, they typically run regression tests to detect if their recent changes (re)introduce any bugs. However, many tests are flaky, and their outcomes can change non-deterministically, failing without apparent cause. Flaky tests are a significant nuisance in the development process, since they make it more difficult for developers to trust the outcome of their tests, and hence, it is important to know which tests are flaky. The traditional approach to identify flaky tests is to rerun them multiple times: if a test is observed both passing and failing on the same code, it is definitely flaky. We conducted a very large empirical study looking for flaky tests by rerunning the test suites of 24 projects 10,000 times each, and found that even with this many reruns, some previously identified flaky tests were still not detected. We propose FlakeFlagger, a novel approach that collects a set of features describing the behavior of each test, and then predicts tests that are likely to be flaky based on similar behavioral features. We found that FlakeFlagger correctly labeled as flaky at least as many tests as a state-of-the-art flaky test classifier, but that FlakeFlagger reported far fewer false positives. This lower false positive rate translates directly to saved time for researchers and developers who use the classification result to guide more expensive flaky test detection processes. Evaluated on our dataset of 23 projects with flaky tests, FlakeFlagger outperformed the prior approach (by F1 score) on 16 projects and tied on 4 projects. Our results indicate that this approach can be effective for identifying likely flaky tests prior to running time-consuming flaky test detectors.

show abstract