A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests

Abstract

Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic outcomes that cripple continuous integration and lead developers to investigate false alerts. Industrial reports indicate that on a large scale, the accrual of flaky tests breaks the trust in test suites and entails significant computational cost. To alleviate this, practitioners are constrained to identify flaky tests and investigate their impact. To shed light on such mitigation mechanisms, we interview 14 practitioners with the aim to identify (i) the sources of flakiness within the testing ecosystem, (ii) the impacts of flakiness, (iii) the measures adopted by practitioners when addressing flakiness, and (iv) the automation opportunities for these measures. Our analysis shows that, besides the tests and code, flakiness stems from interactions between the system components, the testing infrastructure, and external factors. We also highlight the impact of flakiness on testing practices and product quality and show that the adoption of guidelines together with a stable infrastructure are key measures in mitigating the problem.• RQ1: Where can we locate flakiness? Goal: Differently from previous studies [6]-[10], which focused on the root causes of flakiness, e.g., concurrency and timeouts, we aim to identify where flakiness stems within the different components of the development ecosystem, e.g., test, code under test, and infrastructure. This localisation is necessary to guide both detection and fixing approaches. Results: In addition to tests, flakiness stems from the poor orchestration between the system components, the testing infrastructure, and external factors, e.g., OS and firmware. Studies should consider and leverage these factors when addressing flaky tests and not focus solely on the test and code under test.• RQ2: How do practitioners perceive the impact of flakiness? Goal: This question is commonly discussed in industrial reports and research studies. In this paper, we examine it through direct discussions with practitioners. The aim is to understand the impact of flakiness on the development workflow and practices. Results: Besides dissipating development time and hindering the continuous integration (CI), flakiness affects the testing practices and leads to a degradation of the system quality. We also shed light on the pernicious consequences of system flakiness, i.e., buggy or non-deterministic features that are falsely labelled as flaky tests.• RQ3: How do practitioners address flaky tests? Goal: This question aims at identifying and understanding the measures taken by practitioners to address flakiness before and after it manifests in the CI. Results: The prevention of test flakiness is performed by building stable infrastructures and enforcing guidelines, whereas the detection still relies mainly on reruns and manual inspection. Our results also highlight monitoring and logging tasks, which are commonly dismissed in research, yet they are key to most of the mitigation measures taken by practitioners.• RQ4: Ho...