An empirical analysis of flaky tests

Luo, Qingzhou; Hariri, Farah; Eloussi, Lamyaa; Marinov, Darko

doi:10.1145/2635868.2635920

Cited by 302 publications

(315 citation statements)

References 27 publications

(36 reference statements)

Supporting

Mentioning

285

Contrasting

Unclassified

Order By: Relevance

“…This investigation can be considered as a large-scale replication of the study proposed by Luo et al [18], who performed a similar analysis inspecting 201 commits that likely fix flaky tests. With RQ 2 our goal is to perform a fine-grained investigation into the relationship between test smells and flaky tests.…”

Section: Empirical Study Definition and Designmentioning

confidence: 76%

“…These causes have been classified by relying on the taxonomy proposed by Luo et al [18], who identified ten common causes of test flakiness. Table III reports, for each common cause, a brief description.…”

Section: Discussionmentioning

confidence: 99%

“…Some key issues that are associated with the presence of flaky tests are that: (i) they may hide real bugs and are hard to reproduce due to their non-determinism [18], (ii) they increase maintenance costs because developers may have to spend substantial time debugging failures that are not really failures, but just flaky [19]. Perhaps most importantly, from a psychological point of view flaky tests can reduce a developer's confidence in the tests, possibly leading to ignoring actual test failures [17].…”

Section: Introductionmentioning

confidence: 99%

“…Perhaps most importantly, from a psychological point of view flaky tests can reduce a developer's confidence in the tests, possibly leading to ignoring actual test failures [17]. Because of this, the research community has spent considerably effort on trying to understand the causes behind test flakiness [18], [20], [21], [22] and on devising automated techniques able to fix flaky tests [23], [24], [25]. However, most of this research mainly focused the attention on some specific causes possibly leading to the introduction of flaky tests, such as concurrency [26], [25], [27] or test order dependency [22] issues, thus proposing ad-hoc solutions that cannot be used to fix flaky tests characterized by other root causes.…”

Section: Introductionmentioning

confidence: 99%

“…However, most of this research mainly focused the attention on some specific causes possibly leading to the introduction of flaky tests, such as concurrency [26], [25], [27] or test order dependency [22] issues, thus proposing ad-hoc solutions that cannot be used to fix flaky tests characterized by other root causes. Indeed, according to the findings by Luo et al [18] who conducted an empirical study on the motivations behind test code flakiness, the problems faced by previous research only represent a part of whole story: a deeper analysis of possible fixing strategies of other root causes (e.g., flakiness due to wrong usage of external resources) is still missing.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

Palomba

Zaidman

2017

2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

Abstract-Regression testing is a core activity that allows developers to ensure that source code changes do not introduce bugs. An important prerequisite then is that test cases are deterministic. However, this is not always the case as some tests suffer from socalled flakiness. Flaky tests have serious consequences, as they can hide real bugs and increase software inspection costs. Existing research has focused on understanding the root causes of test flakiness and devising techniques to automatically fix flaky tests; a key area of investigation being concurrency. In this paper, we investigate the relationship between flaky tests and three previously defined test smells, namely Resource Optimism, Indirect Testing and Test Run War. We have set up a study involving 19,532 JUnit test methods belonging to 18 software systems. A key result of our investigation is that 54% of tests that are flaky contain a test code smell that can cause the flakiness. Moreover, we found that refactoring the test smells not only removed the design flaws, but also fixed all 54% of flaky tests causally co-occurring with test smells.

show abstract

Section: Empirical Study Definition and Designmentioning

confidence: 76%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

Palomba

Zaidman

2017

2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

show abstract

Work practices and challenges in continuous integration: A survey with Travis CI users

et al. 2018

View full text Add to dashboard Cite

Continuous integration (CI) is a software development practice that has been gaining increasing popularity in the last few years. However, we still miss a collection of experiences regarding how software developers perceive the idea of CI, in terms of its fundamental concepts, the reasons that motivate the adoption of this practice, the reasons for build breakage, and the benefits and problems related to CI. To shed light on this direction, we conducted a user survey with 158 CI users. Through a mostly qualitative investigation, we produce a list of findings that are not always obvious. For instance, we observed that (1) developers are not sure whether a job failure represents a failure or not; (2) inadequate testing is the most common technical reason related to build breakage, whereas lack of time plays a role on the social reasons; and (3) although some respondents reported that CI systems increase the confidence that the code is in a known state, some respondents also reported that there is a false sense of confidence when blindly trusting tests. This empirical study is particularly relevant to those interested in better understanding and fostering CI practices either in an open-source or industrial setting.

show abstract

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software‐defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.

show abstract

An empirical analysis of flaky tests

Cited by 302 publications

References 27 publications

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

Work practices and challenges in continuous integration: A survey with Travis CI users

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

Contact Info

Product

Resources

About