Assessing Transition-Based Test Selection Algorithms at Google

Abstract

Continuous Integration traditionally relies on testing every code commit with all impacted tests. This practice requires considerable computational resources, which at Google scale, results in delayed test results and high operational costs. To deal with this issue and provide fast feedback, test selection and prioritization methods aim to execute the tests which are most likely to reveal changes in test results as soon as possible. In this paper we present a simulation framework to support the study and evaluation, with real data, of such techniques. We propose a test selection algorithm evaluation method, and detail several practical requirements which are often ignored by related work, such as the detection of transitions, the collection and analysis of data, and the handling of flaky tests. Based on this framework, we design an experiment evaluating five potential regression test selection algorithms, based on simple heuristics and inspired by previous research, though the evaluation technique is applicable to any number of algorithms for future experiments. Our results show that algorithms based on the recent (transition) execution history do not perform as well as expected (given the previously reported results) and that the test selection problem remains largely open. We found that the best performing algorithms are based on the number of times a test has been triggered and the number of distinct authors committing code that triggers particular tests. More research is needed in order to close the gap between the current approaches and the optimal solution.

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

“…There are several approaches aiming at flakiness‐aware 55 or flakiness‐preventing test selection 60 . There are a few approaches to the prediction of flaky tests 56 .…”

What Helped, and what did not? An Evaluation of the Strategies to Improve Continuous Integration

“…Zhu et al [6 6 ] propose a regression test selection framework to check the output against rules inspired by existing test suites for three techniques. Leong et al [30] propose a test selection algorithm evaluation method and evaluate five potential regression test selection algorithms, finding that the test selection problem remains largely open. Najafi et al [41] studied the impact of test execution history on test selection and prioritization techniques.…”

Know You Neighbor: Fast Static Prediction of Test Flakiness

“…The authors complemented the DeFlaker dataset, maintaining the information on flaky tests, by rerunning 100 times the test suites of each project in the most recent version present in GitHub at the time of the study; test cases that had a consistent outcome across all executions (e.g., the test passes 100 times) were flagged as non-flaky. The dataset is accessible in a replication package available online 7 . Note that the tests labeled as flaky could come from different versions of each software project [12], while the tests labeled as non-flaky all come from the same version (the last at the time of rerun) of each software project [21].…”

“…Flakiness hinders regression testing in many ways [2]- [5], especially in a Continuous Integration (CI) environment where ideally all tests must pass before a change can be integrated, or in other words any failing test must be fixed before a release. Indeed, in Google, almost 16% of individual tests contain some form of flakiness [6], and these flaky tests are the cause of 84% of all observed transitions (i.e., changes from pass to fail or the vice versa for the test results across project commits) [7]. A non negligible percentage of flaky tests is observed also in Microsoft: while monitoring five projects over a one-month period, 4.6% individual tests were identified as flaky [8].…”