Notice of Retraction: Does Refactoring of Test Smells Induce Fixing Flaky Tests?

Abstract

Abstract-Regression testing is a core activity that allows developers to ensure that source code changes do not introduce bugs. An important prerequisite then is that test cases are deterministic. However, this is not always the case as some tests suffer from socalled flakiness. Flaky tests have serious consequences, as they can hide real bugs and increase software inspection costs. Existing research has focused on understanding the root causes of test flakiness and devising techniques to automatically fix flaky tests; a key area of investigation being concurrency. In this paper, we investigate the relationship between flaky tests and three previously defined test smells, namely Resource Optimism, Indirect Testing and Test Run War. We have set up a study involving 19,532 JUnit test methods belonging to 18 software systems. A key result of our investigation is that 54% of tests that are flaky contain a test code smell that can cause the flakiness. Moreover, we found that refactoring the test smells not only removed the design flaws, but also fixed all 54% of flaky tests causally co-occurring with test smells.

Understanding flaky tests: the developer’s perspective

“…Design. According to our participants, the probability for a test to be flaky can be reduced if good design principles are applied while developing it, thus confirming previous findings in the field [31]. Six developers report that keeping tests decoupled and mocking dependencies are major challenges when designing test to be less prone to flakiness.…”

“…This motivates the growing research area around test code quality [18,29,[33][34][35][36][37] and provides two promising directions that the research community can focus on: (i) the definition of a set of design patterns that can support the creation of deterministic tests; (ii) the definition of a set of flakiness-related anti-patterns that practitioners should avoid when writing test cases. While some initial steps have been done about the relation between test smells and flaky tests [31,32], further investigation is necessary.…”

“…This occurs mostly due to tests not properly cleaning up after themselves (e.g., restoring common states and shared variables) or failing to setup the necessary preconditions (e.g., a test environment state) before starting to run [24]. 11% of the flaky tests are due to this reason, confirming its diffusion [30]. This is, by nature, a problem raising in test code.…”

Not all bugs are the same: Understanding, characterizing, and classifying bug types

“…Indeed, topics such as "error", "file", "patch", "crash", and exception are concerned with problems caused by issues in the logic of the program (e.g., a wrong return value or an exception). Finally, permission/deprecation and test coderelated issues follow the same discussion: all the words extracted by LDA-GA have clearly something to do with their nature: as an example, the word "retry" appearing in tests is connected with a JUnit annotation (@Retry) that highlights the presence of some form of test flakiness, i.e., unreliable tests that exhibit a pass and fail behavior with the same code [61].…”

The Scent of a Smell: An Extensive Comparison Between Textual and Structural Smells

“…Code smells, i.e., symptoms of poor design and implementation choices applied by programmers during the development of a software project [35], represent an important factor contributing to technical debt [53]. The research community spent a lot of effort studying the extent to which code smells tend to remain in a software project for long periods of time [4], [22], [61], [84], as well as their negative impact on non-functional properties of source code, such as program comprehension [1], change-and fault-proneness [48], [49], testability [77], [68] and, more generally, maintainability [97], [111], [109]. As a consequence, several tools and techniques have been proposed to help developers in detecting code smells and to suggest refactoring opportunities [12], [8], [65], [67], [74], [75], [100].…”