Modeling and ranking flaky tests at Apple

Kowalczyk, Emily; Nair, K. Ravindran; Gao, Zebao; Silberstein, Leo; Long, Teng; Memon, Atif M.

doi:10.1145/3377813.3381370

Cited by 35 publications

(23 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years flaky tests are drawing increasing researchers' attention, also triggered by practitioners' alerts about the relevance and spread of the problem [3], [6], [8], [17], [23]. This overview of related work is based on a thorough search of literature.…”

Section: Related Workmentioning

confidence: 99%

“…This overview of related work is based on a thorough search of literature. 3 Many empirical studies have been conducted aiming at better understanding the nature and extent of test flakiness (e.g., [1], [9], [23]- [31]). Both studies by Luo et al [1] and Vahabzadeh et al [24] examined the causes of flaky tests over the central repository of the Apache Software Foundation, whereas Thorve et al [26] conducted a similar study over Android apps.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Know You Neighbor: Fast Static Prediction of Test Flakiness

et al. 2021

View full text Add to dashboard Cite

Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem.Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and k-Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 nonflaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-theart, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion:The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Know You Neighbor: Fast Static Prediction of Test Flakiness

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Several studies and reports from industrial actors have highlighted the prevalence and impact of flakiness [3]- [5]. For instance, at Google, there are 150 million test executions per day, and almost 16% of their 4.2 million test cases have some level of flakiness [6].…”

Section: Introductionmentioning

confidence: 99%

“…Micco and Memon [11] presented a dynamic approach that identifies flaky tests by looking for specific patterns at the test execution outcomes observed in the recent development history (Pass to Fail, Fail to Pass), thereby proposing a simple pattern matching approach that achieves a 90% accuracy in classifying tests [11]. A similar approach was also presented by Apple [5], but the reality is that rerunning tests is still the main dynamic approach used to detect flakiness [12].…”

Section: Introductionmentioning

confidence: 99%

A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests

Haben

Habchi

Papadakis

et al. 2021

2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)

View full text Add to dashboard Cite

Industrial reports indicate that flaky tests are one of the primary concerns of software testing mainly due to the false signals they provide. To deal with this issue, researchers have developed tools and techniques aiming at (automatically) identifying flaky tests with encouraging results. However, to reach industrial adoption and practice, these techniques need to be replicated and evaluated extensively on multiple datasets, occasions and settings. In view of this, we perform a replication study of a recently proposed method that predicts flaky tests based on their vocabulary. We thus replicate the original study on three different dimensions. First, we replicate the approach on the same subjects as in the original study but using a different evaluation methodology, i.e., we adopt a time-sensitive selection of training and test sets to better reflect the envisioned use case. Second, we consolidate the findings of the initial study by building a new dataset of 837 flaky tests from 9 projects in a different programming language, i.e., Python while the original study was in Java, which comforts the generalisability of the results. Third, we propose an extension to the original approach by experimenting with different features extracted from the Code Under Test. We find that a more robust validation consistently decreases performance on the reported results of the original study, but, fortunately, the model remains capable to decently predict flaky tests. We find re-assuring results that the vocabulary-based models can also be used to predict test flakiness in Python. Finally, we find that the information lying in the Code Under Test has a limited impact on the performance of the vocabulary-based models.

show abstract

“…these tests can mislead developers to debug their recent changes while the failures can be due to a variety of reasons unrelated to the changes. Many software organizations have reported flaky tests as one of their biggest problems in software development, including Apple [18], Facebook [5,10], Google [8,30,31,43,48], Huawei [16], Microsoft [11,12,20,21], and Mozilla [40].…”

Section: Introductionmentioning

confidence: 99%

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Wei

Xie

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Software developers frequently check their code changes by running a set of tests against their code. Tests that can nondeterministically pass or fail when run on the same code version are called flaky tests. These tests are a major problem because they can mislead developers to debug their recent code changes when the failures are unrelated to these changes. One prominent category of flaky tests is order-dependent (OD) tests, which can deterministically pass or fail depending on the order in which the set of tests are run. By detecting OD tests in advance, developers can fix these tests before they change their code. Due to the high cost required to explore all possible orders (n! permutations for n tests), prior work has developed tools that randomize orders to detect OD tests. Experiments have shown that randomization can detect many OD tests, and that most OD tests depend on just one other test to fail. However, there was no analysis of the probability that randomized orders detect OD tests. In this paper, we present the first such analysis and also present a simple change for sampling random test orders to increase the probability. We finally present a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test, while running substantially fewer orders and tests than simply running all test pairs.

show abstract

Modeling and ranking flaky tests at Apple

Cited by 35 publications

References 12 publications

Know You Neighbor: Fast Static Prediction of Test Flakiness

Know You Neighbor: Fast Static Prediction of Test Flakiness

A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests

Contact Info

Product

Resources

About